Getting started with Advanced HPO Algorithms

Loading libraries

# Basic utils for folder manipulations etc
import time
import multiprocessing # to count the number of CPUs available

# External tools to load and process data
import numpy as np
import pandas as pd

# MXNet (NeuralNets)
import mxnet as mx
from mxnet import gluon, autograd
from mxnet.gluon import nn

# AutoGluon and HPO tools
import autogluon as ag
from autogluon.utils import load_and_split_openml_data

Check the version of MxNet, you should be fine with version >= 1.5

mx.__version__
'1.6.0'

You can also check the version of AutoGluon and the specific commit and check that it matches what you want.

ag.__version__
'0.0.13b20200806'

Hyperparameter Optimization of a 2-layer MLP

Setting up the context

Here we declare a few “environment variables” setting the context for what we’re doing

OPENML_TASK_ID = 6                # describes the problem we will tackle
RATIO_TRAIN_VALID = 0.33          # split of the training data used for validation
RESOURCE_ATTR_NAME = 'epoch'      # how do we measure resources   (will become clearer further)
REWARD_ATTR_NAME = 'objective'    # how do we measure performance (will become clearer further)

NUM_CPUS = multiprocessing.cpu_count()

Preparing the data

We will use a multi-way classification task from OpenML. Data preparation includes:

  • Missing values are imputed, using the ‘mean’ strategy of sklearn.impute.SimpleImputer

  • Split training set into training and validation

  • Standardize inputs to mean 0, variance 1

X_train, X_valid, y_train, y_valid, n_classes = load_and_split_openml_data(
    OPENML_TASK_ID, RATIO_TRAIN_VALID, download_from_openml=False)
n_classes
Downloading ./org/openml/www/datasets/6/dataset.arff from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.arff...
100%|██████████| 704/704 [00:00<00:00, 4043.69KB/s]
Downloading ./org/openml/www/datasets/6/dataset.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.pkl.py3...
100%|██████████| 2521/2521 [00:00<00:00, 32469.55KB/s]
Downloading ./org/openml/www/datasets/6/description.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/description.xml...
3KB [00:00, 3110.73KB/s]
Downloading ./org/openml/www/datasets/6/features.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/features.xml...
8KB [00:00, 7843.49KB/s]
Downloading ./org/openml/www/datasets/6/qualities.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/qualities.xml...
15KB [00:00, 12871.23KB/s]
Downloading ./org/openml/www/tasks/6/datasplits.arff from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.arff...
2998KB [00:00, 23655.31KB/s]
Downloading ./org/openml/www/tasks/6/datasplits.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.pkl.py3...
881KB [00:00, 61120.83KB/s]
Downloading ./org/openml/www/tasks/6/task.xml from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/task.xml...
3KB [00:00, 3653.57KB/s]
Data pickle file already exists and is up to date.
Data pickle file already exists and is up to date.
26

The problem has 26 classes.

Declaring a model specifying a hyperparameter space with AutoGluon

Two layer MLP where we optimize over:

  • the number of units on the first layer

  • the number of units on the second layer

  • the dropout rate after each layer

  • the learning rate

  • the scaling

  • the @ag.args decorator allows us to specify the space we will optimize over, this matches the ConfigSpace syntax

The body of the function run_mlp_openml is pretty simple:

  • it reads the hyperparameters given via the decorator

  • it defines a 2 layer MLP with dropout

  • it declares a trainer with the ‘adam’ loss function and a provided learning rate

  • it trains the NN with a number of epochs (most of that is boilerplate code from mxnet)

  • the reporter at the end is used to keep track of training history in the hyperparameter optimization

Note: The number of epochs and the hyperparameter space are reduced to make for a shorter experiment

@ag.args(n_units_1=ag.space.Int(lower=16, upper=128),
         n_units_2=ag.space.Int(lower=16, upper=128),
         dropout_1=ag.space.Real(lower=0, upper=.75),
         dropout_2=ag.space.Real(lower=0, upper=.75),
         learning_rate=ag.space.Real(lower=1e-6, upper=1, log=True),
         batch_size=ag.space.Int(lower=8, upper=128),
         scale_1=ag.space.Real(lower=0.001, upper=10, log=True),
         scale_2=ag.space.Real(lower=0.001, upper=10, log=True),
         epochs=9)
def run_mlp_openml(args, reporter, **kwargs):
    # Time stamp for elapsed_time
    ts_start = time.time()
    # Unwrap hyperparameters
    n_units_1 = args.n_units_1
    n_units_2 = args.n_units_2
    dropout_1 = args.dropout_1
    dropout_2 = args.dropout_2
    scale_1 = args.scale_1
    scale_2 = args.scale_2
    batch_size = args.batch_size
    learning_rate = args.learning_rate

    ctx = mx.cpu()
    net = nn.Sequential()
    with net.name_scope():
        # Layer 1
        net.add(nn.Dense(n_units_1, activation='relu',
                         weight_initializer=mx.initializer.Uniform(scale=scale_1)))
        # Dropout
        net.add(gluon.nn.Dropout(dropout_1))
        # Layer 2
        net.add(nn.Dense(n_units_2, activation='relu',
                         weight_initializer=mx.initializer.Uniform(scale=scale_2)))
        # Dropout
        net.add(gluon.nn.Dropout(dropout_2))
        # Output
        net.add(nn.Dense(n_classes))
    net.initialize(ctx=ctx)

    trainer = gluon.Trainer(net.collect_params(), 'adam',
                            {'learning_rate': learning_rate})

    for epoch in range(args.epochs):
        ts_epoch = time.time()

        train_iter = mx.io.NDArrayIter(
                        data={'data': X_train},
                        label={'label': y_train},
                        batch_size=batch_size,
                        shuffle=True)
        valid_iter = mx.io.NDArrayIter(
                        data={'data': X_valid},
                        label={'label': y_valid},
                        batch_size=batch_size,
                        shuffle=False)

        metric = mx.metric.Accuracy()
        loss = gluon.loss.SoftmaxCrossEntropyLoss()

        for batch in train_iter:
            data = batch.data[0].as_in_context(ctx)
            label = batch.label[0].as_in_context(ctx)
            with autograd.record():
                output = net(data)
                L = loss(output, label)
            L.backward()
            trainer.step(data.shape[0])
            metric.update([label], [output])

        name, train_acc = metric.get()

        metric = mx.metric.Accuracy()
        for batch in valid_iter:
            data = batch.data[0].as_in_context(ctx)
            label = batch.label[0].as_in_context(ctx)
            output = net(data)
            metric.update([label], [output])

        name, val_acc = metric.get()

        print('Epoch %d ; Time: %f ; Training: %s=%f ; Validation: %s=%f' % (
            epoch + 1, time.time() - ts_start, name, train_acc, name, val_acc))

        ts_now = time.time()
        eval_time = ts_now - ts_epoch
        elapsed_time = ts_now - ts_start

        # The resource reported back (as 'epoch') is the number of epochs
        # done, starting at 1
        reporter(
            epoch=epoch + 1,
            objective=float(val_acc),
            eval_time=eval_time,
            time_step=ts_now,
            elapsed_time=elapsed_time)

Note: The annotation epochs=9 specifies the maximum number of epochs for training. It becomes available as args.epochs. Importantly, it is also processed by HyperbandScheduler below in order to set its max_t attribute.

Recommendation: Whenever writing training code to be passed as train_fn to a scheduler, if this training code reports a resource (or time) attribute, the corresponding maximum resource value should be included in train_fn.args:

  • If the resource attribute (time_attr of scheduler) in train_fn is epoch, make sure to include epochs=XYZ in the annotation. This allows the scheduler to read max_t from train_fn.args.epochs. This case corresponds to our example here.

  • If the resource attribute is something else than epoch, you can also include the annotation max_t=XYZ, which allows the scheduler to read max_t from train_fn.args.max_t.

Annotating the training function by the correct value for max_t simplifies scheduler creation (since max_t does not have to be passed), and avoids inconsistencies between train_fn and the scheduler.

Running the Hyperparameter Optimization

You can use the following schedulers:

  • FIFO (fifo)

  • Hyperband (either the stopping (hbs) or promotion (hbp) variant)

And the following searchers:

  • Random search (random)

  • Gaussian process based Bayesian optimization (bayesopt)

  • SkOpt Bayesian optimization (skopt; only with FIFO scheduler)

Note that the method known as (asynchronous) Hyperband is using random search. Combining Hyperband scheduling with the bayesopt searcher uses a novel method called asynchronous BOHB.

Pick the combination you’re interested in (doing the full experiment takes around 120 seconds, see the time_out parameter), running everything with multiple runs can take a fair bit of time. In real life, you will want to choose a larger time_out in order to obtain good performance.

SCHEDULER = "hbs"
SEARCHER = "bayesopt"
def compute_error(df):
    return 1.0 - df["objective"]

def compute_runtime(df, start_timestamp):
        return df["time_step"] - start_timestamp

def process_training_history(task_dicts, start_timestamp,
                             runtime_fn=compute_runtime,
                             error_fn=compute_error):
    task_dfs = []
    for task_id in task_dicts:
        task_df = pd.DataFrame(task_dicts[task_id])
        task_df = task_df.assign(task_id=task_id,
                                 runtime=runtime_fn(task_df, start_timestamp),
                                 error=error_fn(task_df),
                                 target_epoch=task_df["epoch"].iloc[-1])
        task_dfs.append(task_df)

    result = pd.concat(task_dfs, axis="index", ignore_index=True, sort=True)
    # re-order by runtime
    result = result.sort_values(by="runtime")
    # calculate incumbent best -- the cumulative minimum of the error.
    result = result.assign(best=result["error"].cummin())
    return result

resources = dict(num_cpus=NUM_CPUS, num_gpus=0)
search_options = {
    'num_init_random': 2,
    'debug_log': True}
if SCHEDULER == 'fifo':
    myscheduler = ag.scheduler.FIFOScheduler(
        run_mlp_openml,
        resource=resources,
        searcher=SEARCHER,
        search_options=search_options,
        time_out=120,
        time_attr=RESOURCE_ATTR_NAME,
        reward_attr=REWARD_ATTR_NAME)

else:
    # This setup uses rung levels at 1, 3, 9 epochs. We just use a single
    # bracket, so this is in fact successive halving (Hyperband would use
    # more than 1 bracket).
    # Also note that since we do not use the max_t argument of
    # HyperbandScheduler, this value is obtained from train_fn.args.epochs.
    sch_type = 'stopping' if SCHEDULER == 'hbs' else 'promotion'
    myscheduler = ag.scheduler.HyperbandScheduler(
        run_mlp_openml,
        resource=resources,
        searcher=SEARCHER,
        search_options=search_options,
        time_out=120,
        time_attr=RESOURCE_ATTR_NAME,
        reward_attr=REWARD_ATTR_NAME,
        type=sch_type,
        grace_period=1,
        reduction_factor=3,
        brackets=1)

# run tasks
myscheduler.run()
myscheduler.join_jobs()

results_df = process_training_history(
                myscheduler.training_history.copy(),
                start_timestamp=myscheduler._start_time)
max_t = 9, as inferred from train_fn.args
scheduler_options: Key 'resume': Imputing default value False
scheduler_options: Key 'keep_size_ratios': Imputing default value False
scheduler_options: Key 'maxt_pending': Imputing default value False
scheduler_options: Key 'searcher_data': Imputing default value rungs
scheduler_options: Key 'do_snapshots': Imputing default value False

scheduler_options: Key 'visualizer': Imputing default value none
scheduler_options: Key 'training_history_callback_delta_secs': Imputing default value 60
scheduler_options: Key 'delay_get_config': Imputing default value True

search_options: Key 'random_seed': Imputing default value 31415927
search_options: Key 'opt_skip_init_length': Imputing default value 150
search_options: Key 'opt_skip_period': Imputing default value 1
search_options: Key 'profiler': Imputing default value False
search_options: Key 'opt_maxiter': Imputing default value 50
search_options: Key 'opt_nstarts': Imputing default value 2
search_options: Key 'opt_warmstart': Imputing default value False
search_options: Key 'opt_verbose': Imputing default value False
search_options: Key 'opt_debug_writer': Imputing default value False
search_options: Key 'num_fantasy_samples': Imputing default value 20
search_options: Key 'num_init_candidates': Imputing default value 250
search_options: Key 'initial_scoring': Imputing default value thompson_indep
search_options: Key 'first_is_default': Imputing default value True
search_options: Key 'opt_skip_num_max_resource': Imputing default value False
search_options: Key 'gp_resource_kernel': Imputing default value matern52
search_options: Key 'resource_acq': Imputing default value bohb

[GPMultiFidelitySearcher.__init__]
- acquisition_class = <class 'autogluon.searcher.bayesopt.models.nphead_acqfunc.EIAcquisitionFunction'>
- local_minimizer_class = <class 'autogluon.searcher.bayesopt.tuning_algorithms.bo_algorithm_components.LBFGSOptimizeAcquisition'>
- num_initial_candidates = 250
- num_initial_random_choices = 2
- initial_scoring = thompson_indep
- first_is_default = True
Starting Experiments
Num of Finished Tasks is 0
Time out (secs) is 120
Starting get_config[random] for config_id 0
Start with default config:
{'batch_size': 68, 'dropout_1': 0.375, 'dropout_2': 0.375, 'learning_rate': 0.001, 'n_units_1': 72, 'n_units_2': 72, 'scale_1': 0.1, 'scale_2': 0.1}
[0: random]
batch_size: 68
dropout_1: 0.375
dropout_2: 0.375
learning_rate: 0.001
n_units_1: 72
n_units_2: 72
scale_1: 0.1
scale_2: 0.1
/var/lib/jenkins/miniconda3/envs/autogluon_docs/lib/python3.7/site-packages/distributed/worker.py:3384: UserWarning: Large object of size 1.30 MB detected in task graph:
  (<function run_mlp_openml at 0x7f2c8d327a70>, {'ar ... sReporter}, [])
Consider scattering large objects ahead of time
with client.scatter to reduce scheduler burden and
keep data on workers

    future = client.submit(func, big_data)    # bad

    big_future = client.scatter(big_data)     # good
    future = client.submit(func, big_future)  # good
  % (format_bytes(len(b)), s)
Epoch 1 ; Time: 0.382855 ; Training: accuracy=0.260079 ; Validation: accuracy=0.531250
Update for config_id 0:1: reward = 0.53125, crit_val = 0.46875
config_id 0: Reaches 1, continues to 3
Epoch 2 ; Time: 0.731057 ; Training: accuracy=0.496365 ; Validation: accuracy=0.655247
Epoch 3 ; Time: 1.078117 ; Training: accuracy=0.559650 ; Validation: accuracy=0.694686
Update for config_id 0:3: reward = 0.6946858288770054, crit_val = 0.30531417112299464
config_id 0: Reaches 3, continues to 9
Epoch 4 ; Time: 1.425796 ; Training: accuracy=0.588896 ; Validation: accuracy=0.711063
Epoch 5 ; Time: 1.908778 ; Training: accuracy=0.609385 ; Validation: accuracy=0.726939
Epoch 6 ; Time: 2.272520 ; Training: accuracy=0.628139 ; Validation: accuracy=0.745321
Epoch 7 ; Time: 2.609782 ; Training: accuracy=0.641193 ; Validation: accuracy=0.750501
Epoch 8 ; Time: 2.948503 ; Training: accuracy=0.653751 ; Validation: accuracy=0.763202
Epoch 9 ; Time: 3.300305 ; Training: accuracy=0.665482 ; Validation: accuracy=0.766043
config_id 0: Terminating evaluation at 9
Update for config_id 0:9: reward = 0.766042780748663, crit_val = 0.23395721925133695
Starting get_config[random] for config_id 1
[1: random]
batch_size: 49
dropout_1: 0.6118880110946948
dropout_2: 0.5692033531564655
learning_rate: 0.00010310483923558212
n_units_1: 126
n_units_2: 93
scale_1: 0.17664689253248303
scale_2: 0.5396711634456218
Epoch 1 ; Time: 0.512763 ; Training: accuracy=0.048005 ; Validation: accuracy=0.111743
config_id 1: Terminating evaluation at 1
Update for config_id 1:1: reward = 0.11174305787888926, crit_val = 0.8882569421211107
Starting get_config[BO] for config_id 2
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4740695831238606
- self.std = 0.25382661537049356
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 4
Current best is [0.46875]
[2: BO] (1 evaluations)
batch_size: 50
dropout_1: 0.21593234798980557
dropout_2: 0.2842099148996821
learning_rate: 0.12426792150054146
n_units_1: 61
n_units_2: 79
scale_1: 9.791854554777688
scale_2: 4.705654410614348
Started BO from (top scorer):
batch_size: 50
dropout_1: 0.21593234798980557
dropout_2: 0.2842099148996821
learning_rate: 0.12426792150054146
n_units_1: 61
n_units_2: 79
scale_1: 9.791854554777688
scale_2: 4.705654410614348
Top score values: [-0.27440679  0.04673507  0.04701008  0.10393977  0.10463477]
Labeled: 0:1, 0:3, 0:9, 1:1. Pending:
Targets: [-0.02095755 -0.66484522 -0.94597     1.63177277]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.7572453035969384, 'kernel_inv_bw1': 0.9003213708321409, 'kernel_inv_bw2': 0.967193412029309, 'kernel_inv_bw3': 3.935156155481977, 'kernel_inv_bw4': 26.153730997146404, 'kernel_inv_bw5': 1.6599386110717311, 'kernel_inv_bw6': 4.378455276759351, 'kernel_inv_bw7': 35.45306859772872, 'kernel_inv_bw8': 1.6274469437661863, 'kernel_covariance_scale': 0.8177941757955258, 'mean_mean_value': 0.3168762072733417}
Epoch 1 ; Time: 0.475681 ; Training: accuracy=0.076116 ; Validation: accuracy=0.045378
config_id 2: Terminating evaluation at 1
Update for config_id 2:1: reward = 0.0453781512605042, crit_val = 0.9546218487394958
Starting get_config[BO] for config_id 3
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5701800362469877
- self.std = 0.2974747675593404
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 5
Current best is [0.46875]
[3: BO] (1 evaluations)
batch_size: 22
dropout_1: 0.6341443733804044
dropout_2: 0.04086583670464042
learning_rate: 0.12818863950369533
n_units_1: 19
n_units_2: 101
scale_1: 0.026860146662962745
scale_2: 4.431236744525342
Started BO from (top scorer):
batch_size: 22
dropout_1: 0.6341443733804044
dropout_2: 0.04086583670464042
learning_rate: 0.12818863950369533
n_units_1: 19
n_units_2: 101
scale_1: 0.026860146662962745
scale_2: 4.431236744525342
Top score values: [-0.02336009  0.00183452  0.04264146  0.12438426  0.12476886]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1. Pending:
Targets: [-0.34097022 -0.89038095 -1.13025659  1.06925676  1.292351  ]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 8.629109443351176, 'kernel_inv_bw1': 1.74250542457921, 'kernel_inv_bw2': 1.9309204792740968, 'kernel_inv_bw3': 1.3547866499369474, 'kernel_inv_bw4': 72.29401709148367, 'kernel_inv_bw5': 4.416632357086148, 'kernel_inv_bw6': 99.98620162991055, 'kernel_inv_bw7': 100.00000000000004, 'kernel_inv_bw8': 1.345177612206158, 'kernel_covariance_scale': 0.7394315650251619, 'mean_mean_value': 0.38000484142269936}
Epoch 1 ; Time: 1.287004 ; Training: accuracy=0.040652 ; Validation: accuracy=0.034007
config_id 3: Terminating evaluation at 1
Update for config_id 3:1: reward = 0.034006734006734006, crit_val = 0.965993265993266
Starting get_config[BO] for config_id 4
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.6361489078713674
- self.std = 0.3090342341807874
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 6
Current best is [0.46875]
[4: BO] (6 evaluations)
batch_size: 110
dropout_1: 0.5744977547648839
dropout_2: 0.7047749526514967
learning_rate: 3.622175990800583e-05
n_units_1: 92
n_units_2: 92
scale_1: 0.066648545659043
scale_2: 0.021088639611731623
Started BO from (top scorer):
batch_size: 110
dropout_1: 0.5744977553473402
dropout_2: 0.7047748951791846
learning_rate: 3.6221732543522435e-05
n_units_1: 92
n_units_2: 92
scale_1: 0.06664854527483652
scale_2: 0.017643835128767108
Top score values: [-0.0169074   0.02032432  0.05571745  0.06113922  0.08197224]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1. Pending:
Targets: [-0.54168403 -1.07054397 -1.30144704  0.81579322  1.03054259  1.06733922]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.0035785923979742124, 'kernel_inv_bw3': 0.002069863224722539, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 4.569906122762831, 'kernel_inv_bw8': 1.3538924996406003, 'kernel_covariance_scale': 0.780966574930137, 'mean_mean_value': -0.06205703671802589}
Epoch 1 ; Time: 0.233732 ; Training: accuracy=0.041983 ; Validation: accuracy=0.114983
Update for config_id 4:1: reward = 0.11498316498316498, crit_val = 0.885016835016835
config_id 4: Reaches 1, continues to 3
Epoch 2 ; Time: 0.457575 ; Training: accuracy=0.067438 ; Validation: accuracy=0.208754
Epoch 3 ; Time: 0.688602 ; Training: accuracy=0.095785 ; Validation: accuracy=0.246633
config_id 4: Terminating evaluation at 3
Update for config_id 4:3: reward = 0.24663299663299662, crit_val = 0.7533670033670034
Starting get_config[BO] for config_id 5
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.6819096607015054
- self.std = 0.28105513303685026
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 8
Current best is [0.46875]
[5: BO] (33 evaluations)
batch_size: 128
dropout_1: 0.0
dropout_2: 0.75
learning_rate: 0.003209138078256018
n_units_1: 128
n_units_2: 16
scale_1: 0.4851811706447043
scale_2: 0.022316347285376906
Started BO from (top scorer):
batch_size: 106
dropout_1: 0.38750926119776946
dropout_2: 0.04489600246106559
learning_rate: 0.002265665895170696
n_units_1: 93
n_units_2: 20
scale_1: 0.4850843918016997
scale_2: 0.03481291816391773
Top score values: [0.17278973 0.18459205 0.1869189  0.19297214 0.21585366]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3. Pending:
Targets: [-0.7584265  -1.33993457 -1.59382409  0.73418791  0.97031563  1.01077537
  0.72265954  0.25424671]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.042529264186120797, 'kernel_inv_bw1': 0.02357832557647553, 'kernel_inv_bw2': 0.03973208528485137, 'kernel_inv_bw3': 6.923567210521894, 'kernel_inv_bw4': 0.037428426239917156, 'kernel_inv_bw5': 0.05885736295947688, 'kernel_inv_bw6': 0.00010000000395442647, 'kernel_inv_bw7': 0.025104636768339714, 'kernel_inv_bw8': 1.3833914683569162, 'kernel_covariance_scale': 0.8507522465523345, 'mean_mean_value': 0.05452222727231742}
Epoch 1 ; Time: 0.191401 ; Training: accuracy=0.110773 ; Validation: accuracy=0.450798
Update for config_id 5:1: reward = 0.4507978723404255, crit_val = 0.5492021276595744
config_id 5: Reaches 1, continues to 3
Epoch 2 ; Time: 0.399945 ; Training: accuracy=0.151974 ; Validation: accuracy=0.518617
Epoch 3 ; Time: 0.581797 ; Training: accuracy=0.161513 ; Validation: accuracy=0.574967
Update for config_id 5:3: reward = 0.574966755319149, crit_val = 0.425033244680851
config_id 5: Reaches 3, continues to 9
Epoch 4 ; Time: 0.769257 ; Training: accuracy=0.172122 ; Validation: accuracy=0.602227
Epoch 5 ; Time: 0.950772 ; Training: accuracy=0.186678 ; Validation: accuracy=0.569980
Epoch 6 ; Time: 1.131077 ; Training: accuracy=0.196217 ; Validation: accuracy=0.640957
Epoch 7 ; Time: 1.312661 ; Training: accuracy=0.200740 ; Validation: accuracy=0.646277
Epoch 8 ; Time: 1.505706 ; Training: accuracy=0.205921 ; Validation: accuracy=0.668218
Epoch 9 ; Time: 1.704916 ; Training: accuracy=0.212993 ; Validation: accuracy=0.672540
config_id 5: Terminating evaluation at 9
Update for config_id 5:9: reward = 0.6725398936170213, crit_val = 0.32746010638297873
Starting get_config[BO] for config_id 6
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.6142702513032225
- self.std = 0.26813178318606906
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 11
Current best is [0.46875]
[6: BO] (10 evaluations)
batch_size: 54
dropout_1: 0.0
dropout_2: 0.17619505106431926
learning_rate: 1.0000000000000004e-06
n_units_1: 102
n_units_2: 98
scale_1: 0.42878232592151666
scale_2: 0.00332059417669923
Started BO from (top scorer):
batch_size: 54
dropout_1: 0.17226648693648097
dropout_2: 0.17978321761415783
learning_rate: 2.8682166768783764e-06
n_units_1: 104
n_units_2: 98
scale_1: 0.4256582753177773
scale_2: 0.0045765439868914945
Top score values: [0.26910219 0.30657945 0.41944929 0.42287908 0.43152957]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9. Pending:
Targets: [-0.54271914 -1.1522546  -1.41838102  1.02183593  1.26934447  1.31175428
  1.00975192  0.51876264 -0.24267218 -0.70576119 -1.06966113]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0029912571288300016, 'kernel_inv_bw1': 0.053822932930128575, 'kernel_inv_bw2': 0.008761955733602828, 'kernel_inv_bw3': 5.99107398159901, 'kernel_inv_bw4': 0.020069365294496683, 'kernel_inv_bw5': 0.009927002027154538, 'kernel_inv_bw6': 0.012215919003796337, 'kernel_inv_bw7': 0.02089175923817651, 'kernel_inv_bw8': 1.2191341554478625, 'kernel_covariance_scale': 0.8403434819864579, 'mean_mean_value': 0.329076417554536}
Epoch 1 ; Time: 0.432395 ; Training: accuracy=0.060764 ; Validation: accuracy=0.076768
config_id 6: Terminating evaluation at 1
Update for config_id 6:1: reward = 0.07676767676767676, crit_val = 0.9232323232323232
Starting get_config[BO] for config_id 7
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.6400170906306476
- self.std = 0.270546353047318
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 12
Current best is [0.46875]
[7: BO] (27 evaluations)
batch_size: 100
dropout_1: 0.4356280405489459
dropout_2: 0.2913279569764319
learning_rate: 0.0010090949832113998
n_units_1: 16
n_units_2: 28
scale_1: 10.0
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 98
dropout_1: 0.46137011527956384
dropout_2: 0.3585507059711988
learning_rate: 0.0007500307333009615
n_units_1: 82
n_units_2: 29
scale_1: 5.647170184313445
scale_2: 0.0018416686798357334
Top score values: [0.33640367 0.34549649 0.3693474  0.39685807 0.40517526]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1. Pending:
Targets: [-0.63304158 -1.23713706 -1.50088836  0.91755017  1.16284975  1.20488106
  0.905574    0.4189667  -0.33567247 -0.79462851 -1.15528072  1.04682702]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.01817329221506528, 'kernel_inv_bw1': 0.015177120649564996, 'kernel_inv_bw2': 0.017523212840445543, 'kernel_inv_bw3': 5.599465675509502, 'kernel_inv_bw4': 0.039005983498097294, 'kernel_inv_bw5': 0.013380403236308833, 'kernel_inv_bw6': 0.039127311735895454, 'kernel_inv_bw7': 0.6501362624588456, 'kernel_inv_bw8': 1.172328518720046, 'kernel_covariance_scale': 0.8289394096816239, 'mean_mean_value': 0.4350393245176951}
Epoch 1 ; Time: 0.245090 ; Training: accuracy=0.151818 ; Validation: accuracy=0.396000
Update for config_id 7:1: reward = 0.396, crit_val = 0.604
config_id 7: Reaches 1, continues to 3
Epoch 2 ; Time: 0.486910 ; Training: accuracy=0.212727 ; Validation: accuracy=0.436000
Epoch 3 ; Time: 0.720034 ; Training: accuracy=0.235868 ; Validation: accuracy=0.481333
config_id 7: Terminating evaluation at 3
Update for config_id 7:3: reward = 0.48133333333333334, crit_val = 0.5186666666666666
Starting get_config[BO] for config_id 8
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.6287765538738883
- self.std = 0.2525015149386042
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 14
Current best is [0.46875]
[8: BO] (45 evaluations)
batch_size: 128
dropout_1: 0.0
dropout_2: 0.75
learning_rate: 0.0020383468158007866
n_units_1: 128
n_units_2: 16
scale_1: 10.0
scale_2: 2.196319028546823
Started BO from (top scorer):
batch_size: 67
dropout_1: 0.16473551439200332
dropout_2: 0.3332012884008315
learning_rate: 0.0025313573378859147
n_units_1: 108
n_units_2: 112
scale_1: 0.005389198200803596
scale_2: 2.4381701654026275
Top score values: [0.28911849 0.35186715 0.35969463 0.41547191 0.42135214]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3. Pending:
Targets: [-0.63376473 -1.28103145 -1.56363155  1.02763894  1.29046867  1.33550372
  1.01480691  0.49342456 -0.31514435 -0.80689935 -1.19332531  1.16615447
 -0.09812438 -0.43607614]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.038837005523123434, 'kernel_inv_bw1': 0.027836144782973124, 'kernel_inv_bw2': 0.031119068717025092, 'kernel_inv_bw3': 4.991188656552736, 'kernel_inv_bw4': 0.034976644009979095, 'kernel_inv_bw5': 0.041701092068418495, 'kernel_inv_bw6': 0.025326558006529706, 'kernel_inv_bw7': 1.315654342341951, 'kernel_inv_bw8': 1.107046708582646, 'kernel_covariance_scale': 0.8772040809522454, 'mean_mean_value': 0.5823012477284242}
Epoch 1 ; Time: 0.200380 ; Training: accuracy=0.069490 ; Validation: accuracy=0.195645
config_id 8: Terminating evaluation at 1
Update for config_id 8:1: reward = 0.19564494680851063, crit_val = 0.8043550531914894
Starting get_config[BO] for config_id 9
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.6404817871617284
- self.std = 0.2478401141563974
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 15
Current best is [0.46875]
[9: BO] (26 evaluations)
batch_size: 122
dropout_1: 0.5937047326888085
dropout_2: 0.37303996058687555
learning_rate: 0.0008351845890106922
n_units_1: 106
n_units_2: 128
scale_1: 10.0
scale_2: 0.026486365357882662
Started BO from (top scorer):
batch_size: 122
dropout_1: 0.5926878354262678
dropout_2: 0.3771519994860407
learning_rate: 0.003289546045796139
n_units_1: 106
n_units_2: 128
scale_1: 7.856450227222442
scale_2: 0.18615906855362926
Top score values: [0.37313461 0.39394155 0.41274531 0.41885784 0.42425513]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1. Pending:
Targets: [-0.69291361 -1.35235419 -1.64026945  0.9997379   1.26751096  1.31339303
  0.98666452  0.45547597 -0.36830059 -0.86930456 -1.26299845  1.14085864
 -0.14719888 -0.49150688  0.66120558]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.018142034227149436, 'kernel_inv_bw1': 0.01360424503439145, 'kernel_inv_bw2': 0.01865090626139601, 'kernel_inv_bw3': 4.710716144212342, 'kernel_inv_bw4': 0.0182185999905586, 'kernel_inv_bw5': 0.021442449668858127, 'kernel_inv_bw6': 0.052111193469014445, 'kernel_inv_bw7': 2.0089750512467703, 'kernel_inv_bw8': 1.079644308068812, 'kernel_covariance_scale': 0.9142565484730996, 'mean_mean_value': 0.6331161028293205}
Epoch 1 ; Time: 0.221219 ; Training: accuracy=0.298559 ; Validation: accuracy=0.632988
Update for config_id 9:1: reward = 0.6329876212780194, crit_val = 0.36701237872198056
config_id 9: Reaches 1, continues to 3
Epoch 2 ; Time: 0.437266 ; Training: accuracy=0.443534 ; Validation: accuracy=0.707929
Epoch 3 ; Time: 0.686439 ; Training: accuracy=0.482530 ; Validation: accuracy=0.711777
Update for config_id 9:3: reward = 0.7117765138842422, crit_val = 0.28822348611575777
config_id 9: Reaches 3, continues to 9
Epoch 4 ; Time: 0.900390 ; Training: accuracy=0.504471 ; Validation: accuracy=0.730345
Epoch 5 ; Time: 1.138067 ; Training: accuracy=0.511095 ; Validation: accuracy=0.745065
Epoch 6 ; Time: 1.348499 ; Training: accuracy=0.538914 ; Validation: accuracy=0.752760
Epoch 7 ; Time: 1.612757 ; Training: accuracy=0.544958 ; Validation: accuracy=0.772666
Epoch 8 ; Time: 1.825762 ; Training: accuracy=0.552741 ; Validation: accuracy=0.777852
Epoch 9 ; Time: 2.034816 ; Training: accuracy=0.560772 ; Validation: accuracy=0.774005
config_id 9: Terminating evaluation at 9
Update for config_id 9:9: reward = 0.7740046838407494, crit_val = 0.22599531615925061
Starting get_config[BO] for config_id 10
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5826921104679397
- self.std = 0.26161121007299293
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 18
Current best is [0.36701238]
[10: BO] (8 evaluations)
batch_size: 35
dropout_1: 0.4189316406676796
dropout_2: 0.64785944140762
learning_rate: 0.0004923229512480123
n_units_1: 92
n_units_2: 107
scale_1: 0.06187773027317159
scale_2: 0.014919426971865563
Started BO from (top scorer):
batch_size: 35
dropout_1: 0.4189316770736182
dropout_2: 0.647859383861638
learning_rate: 0.0004840516285227221
n_units_1: 92
n_units_2: 107
scale_1: 0.06187776130090886
scale_2: 0.03166162310123019
Top score values: [0.33639694 0.36297432 0.38126212 0.38645803 0.39695491]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9. Pending:
Targets: [-0.43553986 -1.06026779 -1.33302732  1.16801123  1.42168884  1.4651557
  1.15562603  0.652399   -0.12801433 -0.60264568 -0.9756157   1.30170344
  0.08144869 -0.24473509  0.8472991  -0.82442848 -1.12559635 -1.36346143]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0012528653216046406, 'kernel_inv_bw1': 0.000351239318362345, 'kernel_inv_bw2': 0.00045058044207702105, 'kernel_inv_bw3': 6.68223995627897, 'kernel_inv_bw4': 0.0015479203428939742, 'kernel_inv_bw5': 0.0007301684228847166, 'kernel_inv_bw6': 0.0003142868289391012, 'kernel_inv_bw7': 2.612446594927186, 'kernel_inv_bw8': 1.2880199140786603, 'kernel_covariance_scale': 0.78600469004484, 'mean_mean_value': 0.69320155558555}
Epoch 1 ; Time: 0.679502 ; Training: accuracy=0.236605 ; Validation: accuracy=0.500504
Update for config_id 10:1: reward = 0.5005042016806722, crit_val = 0.4994957983193278
config_id 10: Reaches 1, continues to 3
Epoch 2 ; Time: 1.356208 ; Training: accuracy=0.413085 ; Validation: accuracy=0.582521
Epoch 3 ; Time: 2.012082 ; Training: accuracy=0.487619 ; Validation: accuracy=0.626387
config_id 10: Terminating evaluation at 3
Update for config_id 10:3: reward = 0.6263865546218488, crit_val = 0.3736134453781512
Starting get_config[BO] for config_id 11
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5680783616060197
- self.std = 0.2528133635392197
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 20
Current best is [0.36701238]
[11: BO] (27 evaluations)
batch_size: 128
dropout_1: 0.48567407913313465
dropout_2: 0.41288933361639724
learning_rate: 0.002041995112130015
n_units_1: 36
n_units_2: 128
scale_1: 5.784232732906726
scale_2: 0.025972116606279762
Started BO from (top scorer):
batch_size: 105
dropout_1: 0.4856686881223919
dropout_2: 0.41288942543415097
learning_rate: 0.015537307956329766
n_units_1: 36
n_units_2: 118
scale_1: 5.784225726298944
scale_2: 0.1753674900097659
Top score values: [0.28009922 0.28997111 0.3217382  0.34170678 0.35464527]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3. Pending:
Targets: [-0.39289205 -1.03936037 -1.32161187  1.26646225  1.52896778  1.57394727
  1.25364605  0.73290683 -0.0746647  -0.56581312 -0.95176241  1.40480692
  0.14208758 -0.19544732  0.93458941 -0.7953139  -1.10696235 -1.35310508
 -0.27127744 -0.76920347]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.19035005072870462, 'kernel_inv_bw1': 0.0011561453841036151, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 4.326774650479719, 'kernel_inv_bw4': 0.00024049852257538023, 'kernel_inv_bw5': 0.5778725631767939, 'kernel_inv_bw6': 0.000175147876436982, 'kernel_inv_bw7': 1.6193948079704306, 'kernel_inv_bw8': 1.1099815661552153, 'kernel_covariance_scale': 0.8683275627500506, 'mean_mean_value': 0.7830009055909884}
Epoch 1 ; Time: 0.197887 ; Training: accuracy=0.278372 ; Validation: accuracy=0.572806
Update for config_id 11:1: reward = 0.5728058510638298, crit_val = 0.42719414893617025
config_id 11: Reaches 1, continues to 3
Epoch 2 ; Time: 0.406101 ; Training: accuracy=0.363816 ; Validation: accuracy=0.621011
Epoch 3 ; Time: 0.594057 ; Training: accuracy=0.400905 ; Validation: accuracy=0.642287
config_id 11: Terminating evaluation at 3
Update for config_id 11:3: reward = 0.6422872340425532, crit_val = 0.35771276595744683
Starting get_config[BO] for config_id 12
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5521124612279095
- self.std = 0.24650151773873644
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 22
Current best is [0.36701238]
[12: BO] (30 evaluations)
batch_size: 128
dropout_1: 0.3949306838852321
dropout_2: 0.37421973461517766
learning_rate: 0.0008619875734616166
n_units_1: 32
n_units_2: 60
scale_1: 0.19191121634570388
scale_2: 0.024011044036543073
Started BO from (top scorer):
batch_size: 72
dropout_1: 0.39493055316577974
dropout_2: 0.37424420255786994
learning_rate: 0.0015053635333599907
n_units_1: 32
n_units_2: 65
scale_1: 0.1919113363111212
scale_2: 0.02258804603989569
Top score values: [0.39569149 0.41432932 0.43900397 0.44850582 0.46284489]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3. Pending:
Targets: [-0.33818234 -1.00120394 -1.29068269  1.3636609   1.63288807  1.6790193
  1.35051653  0.81644342 -0.01180655 -0.51553117 -0.91136297  1.50554798
  0.21049582 -0.1356819   1.02329022 -0.75090849 -1.07053692 -1.3229823
 -0.21345371 -0.72412948 -0.50676488 -0.78863488]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.2923585253961906, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.0008488941270236102, 'kernel_inv_bw3': 4.8085961725692075, 'kernel_inv_bw4': 0.0009394039000798733, 'kernel_inv_bw5': 0.28904663988794216, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 1.8569050490473502, 'kernel_inv_bw8': 1.24560644192412, 'kernel_covariance_scale': 0.8430521769516296, 'mean_mean_value': 0.8725688626622445}
Epoch 1 ; Time: 0.196646 ; Training: accuracy=0.164145 ; Validation: accuracy=0.238863
config_id 12: Terminating evaluation at 1
Update for config_id 12:1: reward = 0.2388630319148936, crit_val = 0.7611369680851063
Starting get_config[BO] for config_id 13
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5612004832651789
- self.std = 0.24482270307015344
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 23
Current best is [0.36701238]
[13: BO] (31 evaluations)
batch_size: 85
dropout_1: 0.517318414125913
dropout_2: 0.3503723017238424
learning_rate: 0.004339501360450686
n_units_1: 127
n_units_2: 120
scale_1: 0.00503374773788621
scale_2: 0.04682843950719627
Started BO from (top scorer):
batch_size: 85
dropout_1: 0.5173205997487473
dropout_2: 0.3503395161012028
learning_rate: 0.17877383202727398
n_units_1: 21
n_units_2: 38
scale_1: 0.005033730493003303
scale_2: 0.004129261303025423
Top score values: [0.21522201 0.24871193 0.24959443 0.2818383  0.29216209]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1. Pending:
Targets: [-0.37762218 -1.04519029 -1.33665408  1.33589105  1.60696439  1.65341195
  1.32265655  0.78492116 -0.04900835 -0.55618714 -0.95473326  1.47875109
  0.17481841 -0.17373314  0.99318636 -0.7931785  -1.11499871 -1.36917517
 -0.25203825 -0.76621586 -0.54736073 -0.83116359  0.81665827]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.0018009065340422222, 'kernel_inv_bw2': 0.00526751979250695, 'kernel_inv_bw3': 3.6751566246702496, 'kernel_inv_bw4': 0.6931462824479084, 'kernel_inv_bw5': 0.9862999198786682, 'kernel_inv_bw6': 0.0020835114772087687, 'kernel_inv_bw7': 3.171923971732715, 'kernel_inv_bw8': 0.9914290006573513, 'kernel_covariance_scale': 1.0347136339005742, 'mean_mean_value': 0.6951370896269443}
Epoch 1 ; Time: 0.315958 ; Training: accuracy=0.436205 ; Validation: accuracy=0.676303
Update for config_id 13:1: reward = 0.6763025210084034, crit_val = 0.3236974789915966
config_id 13: Reaches 1, continues to 3
Epoch 2 ; Time: 0.610084 ; Training: accuracy=0.617978 ; Validation: accuracy=0.755630
Epoch 3 ; Time: 0.893151 ; Training: accuracy=0.671914 ; Validation: accuracy=0.776807
Update for config_id 13:3: reward = 0.7768067226890756, crit_val = 0.22319327731092442
config_id 13: Reaches 3, continues to 9
Epoch 4 ; Time: 1.177485 ; Training: accuracy=0.701574 ; Validation: accuracy=0.809580
Epoch 5 ; Time: 1.472171 ; Training: accuracy=0.721541 ; Validation: accuracy=0.833445
Epoch 6 ; Time: 1.761903 ; Training: accuracy=0.737200 ; Validation: accuracy=0.837311
Epoch 7 ; Time: 2.053310 ; Training: accuracy=0.749544 ; Validation: accuracy=0.849076
Epoch 8 ; Time: 2.343127 ; Training: accuracy=0.753853 ; Validation: accuracy=0.860840
Epoch 9 ; Time: 2.634812 ; Training: accuracy=0.761060 ; Validation: accuracy=0.865546
config_id 13: Terminating evaluation at 9
Update for config_id 13:9: reward = 0.865546218487395, crit_val = 0.134453781512605
Starting get_config[BO] for config_id 14
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5226521404967017
- self.std = 0.2551554231756705
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 26
Current best is [0.22319328]
[14: BO] (27 evaluations)
batch_size: 8
dropout_1: 0.44083006806595904
dropout_2: 0.5666285599135659
learning_rate: 0.0039505467062455855
n_units_1: 128
n_units_2: 83
scale_1: 0.004122091602642879
scale_2: 0.06906901108503602
Started BO from (top scorer):
batch_size: 89
dropout_1: 0.44083011379014075
dropout_2: 0.5666285180710813
learning_rate: 0.056261366616198794
n_units_1: 74
n_units_2: 70
scale_1: 0.004122085994468912
scale_2: 0.07838948698811744
Top score values: [0.08943119 0.09851775 0.15949493 0.16826384 0.18186243]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9. Pending:
Targets: [-0.21125218 -0.8517866  -1.13144733  1.43287098  1.69296699  1.73753362
  1.42017242  0.90421305  0.10405418 -0.38258601 -0.76499269  1.56994579
  0.31881689 -0.01561979  1.10404439 -0.60998022 -0.91876806 -1.16265146
 -0.09075387 -0.58410945 -0.37411704 -0.64642708  0.93466494 -0.77973911
 -1.17363315 -1.5214192 ]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.19661690687917155, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 2.76708380111987, 'kernel_inv_bw4': 0.45302302511801334, 'kernel_inv_bw5': 1.0006757249815805, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 3.4402454237796487, 'kernel_inv_bw8': 0.9099949398435804, 'kernel_covariance_scale': 1.0560090446899353, 'mean_mean_value': 0.7294833011248456}
Epoch 1 ; Time: 2.904946 ; Training: accuracy=0.453830 ; Validation: accuracy=0.709287
Update for config_id 14:1: reward = 0.7092866756393001, crit_val = 0.29071332436069985
config_id 14: Reaches 1, continues to 3
Epoch 2 ; Time: 5.933223 ; Training: accuracy=0.579078 ; Validation: accuracy=0.756057
Epoch 3 ; Time: 8.843219 ; Training: accuracy=0.601376 ; Validation: accuracy=0.781965
Update for config_id 14:3: reward = 0.7819650067294751, crit_val = 0.2180349932705249
config_id 14: Reaches 3, continues to 9
Epoch 4 ; Time: 11.760121 ; Training: accuracy=0.626824 ; Validation: accuracy=0.795929
Epoch 5 ; Time: 14.631067 ; Training: accuracy=0.638677 ; Validation: accuracy=0.790377
Epoch 6 ; Time: 17.486610 ; Training: accuracy=0.652188 ; Validation: accuracy=0.807032
Epoch 7 ; Time: 20.348637 ; Training: accuracy=0.654095 ; Validation: accuracy=0.820659
Epoch 8 ; Time: 23.204343 ; Training: accuracy=0.665202 ; Validation: accuracy=0.827221
Epoch 9 ; Time: 26.046952 ; Training: accuracy=0.670590 ; Validation: accuracy=0.822678
config_id 14: Terminating evaluation at 9
Update for config_id 14:9: reward = 0.822678331090175, crit_val = 0.17732166890982504
Starting get_config[BO] for config_id 15
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4922422634294929
- self.std = 0.2580920603410797
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 29
Current best is [0.21803499]
[15: BO] (32 evaluations)
batch_size: 28
dropout_1: 0.7067660247935745
dropout_2: 0.5077105068694533
learning_rate: 0.0060571749044140925
n_units_1: 128
n_units_2: 124
scale_1: 8.648184446960908
scale_2: 0.007892685448944886
Started BO from (top scorer):
batch_size: 28
dropout_1: 0.7067660233501636
dropout_2: 0.5077105012206583
learning_rate: 0.6552783935929372
n_units_1: 25
n_units_2: 26
scale_1: 8.64818447109396
scale_2: 0.001987907158095328
Top score values: [0.08101922 0.18297088 0.22671446 0.24364789 0.26306334]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9. Pending:
Targets: [-0.09102281 -0.72426905 -1.00074773  1.53439311  1.79152968  1.83558922
  1.52183903  1.01175038  0.22069592 -0.26040715 -0.63846271  1.66990825
  0.43301501  0.10238363  1.20930799 -0.48521401 -0.79048839 -1.03159681
  0.02810445 -0.45963761 -0.25203454 -0.52124617  1.04185578 -0.65304134
 -1.04245356 -1.3862824  -0.7808413  -1.06243977 -1.22018707]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0017239593234710502, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 3.570434407737034, 'kernel_inv_bw4': 0.7283941471136309, 'kernel_inv_bw5': 1.000855396404901, 'kernel_inv_bw6': 0.00010423808122954203, 'kernel_inv_bw7': 3.232910339204173, 'kernel_inv_bw8': 1.0262492248060313, 'kernel_covariance_scale': 0.8876604836153849, 'mean_mean_value': 0.8701089467021575}
Epoch 1 ; Time: 0.843483 ; Training: accuracy=0.170451 ; Validation: accuracy=0.267606
config_id 15: Terminating evaluation at 1
Update for config_id 15:1: reward = 0.2676056338028169, crit_val = 0.7323943661971831
Starting get_config[BO] for config_id 16
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5002473335217492
- self.std = 0.257389745636864
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 30
Current best is [0.21803499]
[16: BO] (44 evaluations)
batch_size: 8
dropout_1: 0.6233223085212575
dropout_2: 0.4112965755062492
learning_rate: 0.001956180614449219
n_units_1: 96
n_units_2: 128
scale_1: 0.08489975369264192
scale_2: 0.004798678017280562
Started BO from (top scorer):
batch_size: 94
dropout_1: 0.6232277607609598
dropout_2: 0.41130154566559785
learning_rate: 0.009744280412612874
n_units_1: 67
n_units_2: 64
scale_1: 0.47024490449176665
scale_2: 0.002261738493914154
Top score values: [0.12222099 0.18662739 0.19128603 0.20603738 0.21008631]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9, 15:1. Pending:
Targets: [-0.12237214 -0.75734626 -1.03457934  1.50747889  1.76531708  1.80949684
  1.49489056  0.98341008  0.19019714 -0.29221867 -0.67130579  1.6433638
  0.40309557  0.07156203  1.18150674 -0.51763894 -0.82374629 -1.0655126
 -0.00291983 -0.49199275 -0.28382321 -0.55376941  1.01359762 -0.6859242
 -1.07639897 -1.42116599 -0.81407287 -1.09643972 -1.25461744  0.90192806]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.02072833578446393, 'kernel_inv_bw1': 0.0007070265218982463, 'kernel_inv_bw2': 0.00010601560063883542, 'kernel_inv_bw3': 5.71608459837552, 'kernel_inv_bw4': 1.4945779216991482, 'kernel_inv_bw5': 0.3867168213153175, 'kernel_inv_bw6': 1.2627144379756745, 'kernel_inv_bw7': 0.6569959505220573, 'kernel_inv_bw8': 0.9141003039474879, 'kernel_covariance_scale': 1.0905645951879839, 'mean_mean_value': 0.912365266777159}
Epoch 1 ; Time: 2.928490 ; Training: accuracy=0.429377 ; Validation: accuracy=0.677322
Update for config_id 16:1: reward = 0.677321668909825, crit_val = 0.32267833109017496
config_id 16: Reaches 1, continues to 3
Epoch 2 ; Time: 5.862903 ; Training: accuracy=0.556283 ; Validation: accuracy=0.738728
Epoch 3 ; Time: 8.759304 ; Training: accuracy=0.598060 ; Validation: accuracy=0.759253
Update for config_id 16:3: reward = 0.7592530282637954, crit_val = 0.24074697173620463
config_id 16: Reaches 3, continues to 9
Epoch 4 ; Time: 11.696051 ; Training: accuracy=0.622265 ; Validation: accuracy=0.774226
Epoch 5 ; Time: 14.586959 ; Training: accuracy=0.636771 ; Validation: accuracy=0.782806
Epoch 6 ; Time: 17.562423 ; Training: accuracy=0.652105 ; Validation: accuracy=0.789367
Epoch 7 ; Time: 20.509822 ; Training: accuracy=0.653017 ; Validation: accuracy=0.811743
Epoch 8 ; Time: 23.462648 ; Training: accuracy=0.655338 ; Validation: accuracy=0.807201
Epoch 9 ; Time: 26.514607 ; Training: accuracy=0.670093 ; Validation: accuracy=0.815949
config_id 16: Terminating evaluation at 9
Update for config_id 16:9: reward = 0.8159488559892328, crit_val = 0.18405114401076716
Starting get_config[BO] for config_id 17
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4774211046208977
- self.std = 0.2563816663344639
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 33
Current best is [0.21803499]
[17: BO] (36 evaluations)
batch_size: 128
dropout_1: 0.24051539331560012
dropout_2: 0.6439217787214787
learning_rate: 0.00403615530668573
n_units_1: 106
n_units_2: 61
scale_1: 0.01043239707029727
scale_2: 0.0485385641688307
Started BO from (top scorer):
batch_size: 99
dropout_1: 0.2403693569347366
dropout_2: 0.643921725129281
learning_rate: 0.06780717783001759
n_units_1: 110
n_units_2: 84
scale_1: 0.01619971067719354
scale_2: 0.04848851707519957
Top score values: [9.18103952e-05 1.07671531e-01 1.51050172e-01 1.67672935e-01
 2.31863384e-01]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9, 15:1, 16:1, 16:3, 16:9. Pending:
Targets: [-0.03382108 -0.67129189 -0.94961504  1.60243844  1.86129044  1.90564391
  1.58980061  1.07630901  0.27997721 -0.20433544 -0.58491311  1.73885764
  0.49371274  0.16087563  1.27518458 -0.43064205 -0.73795299 -0.98066992
  0.08610091 -0.40489502 -0.19590697 -0.46691458  1.10661526 -0.59958899
 -0.99159909 -1.33772172 -0.72824154 -1.01171864 -1.17051831  0.99450661
 -0.60356411 -0.92313205 -1.14427043]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.023086313292617164, 'kernel_inv_bw1': 0.004640112704300706, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 4.655431728543398, 'kernel_inv_bw4': 1.5147034320525727, 'kernel_inv_bw5': 0.4950980358598371, 'kernel_inv_bw6': 1.3729014795091468, 'kernel_inv_bw7': 0.005020068157930614, 'kernel_inv_bw8': 0.8831416045466082, 'kernel_covariance_scale': 1.147014605226413, 'mean_mean_value': 0.9577840686516704}
Epoch 1 ; Time: 0.209000 ; Training: accuracy=0.286266 ; Validation: accuracy=0.607048
Update for config_id 17:1: reward = 0.6070478723404256, crit_val = 0.39295212765957444
config_id 17: Reaches 1, continues to 3
Epoch 2 ; Time: 0.421269 ; Training: accuracy=0.491118 ; Validation: accuracy=0.693152
Epoch 3 ; Time: 0.671414 ; Training: accuracy=0.553043 ; Validation: accuracy=0.723903
Update for config_id 17:3: reward = 0.7239029255319149, crit_val = 0.27609707446808507
config_id 17: Reaches 3, continues to 9
Epoch 4 ; Time: 0.879142 ; Training: accuracy=0.594901 ; Validation: accuracy=0.749169
Epoch 5 ; Time: 1.067540 ; Training: accuracy=0.603783 ; Validation: accuracy=0.771609
Epoch 6 ; Time: 1.256048 ; Training: accuracy=0.630099 ; Validation: accuracy=0.785073
Epoch 7 ; Time: 1.444981 ; Training: accuracy=0.646135 ; Validation: accuracy=0.801862
Epoch 8 ; Time: 1.639065 ; Training: accuracy=0.657237 ; Validation: accuracy=0.813664
Epoch 9 ; Time: 1.833309 ; Training: accuracy=0.673026 ; Validation: accuracy=0.820977
config_id 17: Terminating evaluation at 9
Update for config_id 17:9: reward = 0.8209773936170213, crit_val = 0.17902260638297873
Starting get_config[BO] for config_id 18
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4611935628055629
- self.std = 0.25256294519747646
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 36
Current best is [0.21803499]
[18: BO] (41 evaluations)
batch_size: 105
dropout_1: 0.75
dropout_2: 0.19746761799270035
learning_rate: 0.0010894554312626916
n_units_1: 77
n_units_2: 128
scale_1: 10.0
scale_2: 0.002951115158859443
Started BO from (top scorer):
batch_size: 126
dropout_1: 0.14441139566675065
dropout_2: 0.1974703086504215
learning_rate: 0.0007212733277897834
n_units_1: 71
n_units_2: 110
scale_1: 0.4701081990937349
scale_2: 0.022516111383238842
Top score values: [0.15113935 0.1618521  0.20883571 0.23071878 0.23280613]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9, 15:1, 16:1, 16:3, 16:9, 17:1, 17:3, 17:9. Pending:
Targets: [ 0.02991903 -0.61719027 -0.89972163  1.69091859  1.9536844   1.99870849
  1.67808968  1.15683415  0.3484619  -0.14317349 -0.52950545  1.82940043
  0.56542909  0.22755953  1.3587167  -0.37290183 -0.68485928 -0.93124606
  0.15165422 -0.34676551 -0.13461759 -0.4097228   1.18759862 -0.54440323
 -0.94234047 -1.29369643 -0.67500099 -0.96276423 -1.12396493  1.0737949
 -0.54843846 -0.87283822 -1.09732019 -0.27019575 -0.7328727  -1.11723023]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.20953941985620783, 'kernel_inv_bw1': 0.048192726740054884, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 4.582304758795538, 'kernel_inv_bw4': 1.4502331780737872, 'kernel_inv_bw5': 0.06676427918254801, 'kernel_inv_bw6': 1.1459117819421119, 'kernel_inv_bw7': 0.7880349587551732, 'kernel_inv_bw8': 0.8158791108576938, 'kernel_covariance_scale': 1.2586479096561005, 'mean_mean_value': 1.094690485256033}
Epoch 1 ; Time: 0.243295 ; Training: accuracy=0.276190 ; Validation: accuracy=0.582623
Update for config_id 18:1: reward = 0.5826232247284879, crit_val = 0.41737677527151207
config_id 18: Reaches 1, continues to 3
Epoch 2 ; Time: 0.479510 ; Training: accuracy=0.328033 ; Validation: accuracy=0.615706
Epoch 3 ; Time: 0.710172 ; Training: accuracy=0.343354 ; Validation: accuracy=0.640936
config_id 18: Terminating evaluation at 3
Update for config_id 18:3: reward = 0.6409356725146199, crit_val = 0.3590643274853801
Starting get_config[BO] for config_id 19
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4573528779936094
- self.std = 0.24645695926305125
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 38
Current best is [0.21803499]
[19: BO] (36 evaluations)
batch_size: 8
dropout_1: 0.75
dropout_2: 0.39774784778258354
learning_rate: 0.001832140609847963
n_units_1: 114
n_units_2: 128
scale_1: 0.01076740318505519
scale_2: 0.047239059463260964
Started BO from (top scorer):
batch_size: 11
dropout_1: 0.27773647423520337
dropout_2: 0.3977763768050508
learning_rate: 0.0011784010178056816
n_units_1: 117
n_units_2: 92
scale_1: 0.0025456093439067047
scale_2: 0.04677717904168564
Top score values: [0.05222215 0.16571061 0.18450087 0.1997315  0.20469509]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9, 15:1, 16:1, 16:3, 16:9, 17:1, 17:3, 17:9, 18:1, 18:3. Pending:
Targets: [ 0.04624386 -0.6168976  -0.90642869  1.74839479  2.01767064  2.06381021
  1.73524805  1.20107838  0.37267866 -0.13113703 -0.52704039  1.89030753
  0.59502123  0.24878092  1.40796258 -0.3665569  -0.68624312 -0.93873414
  0.17099505 -0.33977305 -0.12236915 -0.40429011  1.23260504 -0.54230726
 -0.95010342 -1.31016425 -0.67614059 -0.97103318 -1.13622764  1.11598183
 -0.54644246 -0.87887924 -1.10892277 -0.26130628 -0.73544608 -1.12932608
 -0.16220318 -0.39880615]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.21829910405242756, 'kernel_inv_bw1': 0.025265367905435814, 'kernel_inv_bw2': 0.00018208945300286777, 'kernel_inv_bw3': 5.645833577530412, 'kernel_inv_bw4': 1.0626331487883236, 'kernel_inv_bw5': 0.24181265317433792, 'kernel_inv_bw6': 1.0525747449661047, 'kernel_inv_bw7': 1.2753127010118737, 'kernel_inv_bw8': 0.8183279586931533, 'kernel_covariance_scale': 1.182063180806255, 'mean_mean_value': 1.1579050627358658}
Epoch 1 ; Time: 2.869109 ; Training: accuracy=0.377072 ; Validation: accuracy=0.648890
Update for config_id 19:1: reward = 0.6488896366083445, crit_val = 0.35111036339165547
config_id 19: Reaches 1, continues to 3
Epoch 2 ; Time: 5.864483 ; Training: accuracy=0.502984 ; Validation: accuracy=0.706427
Epoch 3 ; Time: 8.775151 ; Training: accuracy=0.536638 ; Validation: accuracy=0.730989
Update for config_id 19:3: reward = 0.7309892328398385, crit_val = 0.2690107671601615
config_id 19: Reaches 3, continues to 9
Epoch 4 ; Time: 11.672490 ; Training: accuracy=0.560262 ; Validation: accuracy=0.741252
Epoch 5 ; Time: 14.604653 ; Training: accuracy=0.575348 ; Validation: accuracy=0.761945
Epoch 6 ; Time: 17.541553 ; Training: accuracy=0.587782 ; Validation: accuracy=0.756898
Epoch 7 ; Time: 20.415594 ; Training: accuracy=0.594910 ; Validation: accuracy=0.765478
Epoch 8 ; Time: 23.268165 ; Training: accuracy=0.596237 ; Validation: accuracy=0.774058
Epoch 9 ; Time: 26.155721 ; Training: accuracy=0.597066 ; Validation: accuracy=0.774899
config_id 19: Terminating evaluation at 9
Update for config_id 19:9: reward = 0.7748990578734859, crit_val = 0.2251009421265141
Starting get_config[BO] for config_id 20
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4445032057667191
- self.std = 0.24204871271645806
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 41
Current best is [0.13445378]
[20: BO] (33 evaluations)
batch_size: 128
dropout_1: 0.43608638717208853
dropout_2: 0.5726251067457891
learning_rate: 0.002088478516714575
n_units_1: 121
n_units_2: 128
scale_1: 0.021011121096350963
scale_2: 0.006870163092014335
Started BO from (top scorer):
batch_size: 26
dropout_1: 0.37471147279126216
dropout_2: 0.5782873531954602
learning_rate: 0.0006132144475527039
n_units_1: 128
n_units_2: 76
scale_1: 0.762216918632655
scale_2: 0.006595249532295266
Top score values: [0.04338868 0.07714762 0.09796855 0.14017555 0.14090854]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9, 15:1, 16:1, 16:3, 16:9, 17:1, 17:3, 17:9, 18:1, 18:3, 19:1, 19:3, 19:9. Pending:
Targets: [ 0.1001732  -0.57504555 -0.86984964  1.83332409  2.10750405  2.15448392
  1.81993791  1.27603983  0.4325531  -0.08043819 -0.48355184  1.97782137
  0.65894502  0.30639891  1.48669185 -0.32014559 -0.645654   -0.90274345
  0.22719639 -0.29287394 -0.07151063 -0.358566    1.30814066 -0.49909675
 -0.91431979 -1.28093813 -0.63536748 -0.93563072 -1.10383374  1.18939348
 -0.50330726 -0.84179846 -1.07603159 -0.21297811 -0.69575305 -1.09680649
 -0.11207013 -0.35298216 -0.38584317 -0.72502942 -0.90643847]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.24140631994923895, 'kernel_inv_bw1': 0.8134641729505723, 'kernel_inv_bw2': 0.0070302629600143204, 'kernel_inv_bw3': 5.8639041470113975, 'kernel_inv_bw4': 1.1427048768692314, 'kernel_inv_bw5': 0.05156797663092838, 'kernel_inv_bw6': 1.0219083373997764, 'kernel_inv_bw7': 0.7573163628088571, 'kernel_inv_bw8': 0.8114423032606065, 'kernel_covariance_scale': 1.133285180700306, 'mean_mean_value': 1.358492378145194}
Epoch 1 ; Time: 0.231734 ; Training: accuracy=0.292763 ; Validation: accuracy=0.565326
config_id 20: Terminating evaluation at 1
Update for config_id 20:1: reward = 0.5653257978723404, crit_val = 0.4346742021276596
Starting get_config[BO] for config_id 21
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.44426918187055103
- self.std = 0.2391545157785912
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 42
Current best is [0.13445378]
[21: BO] (29 evaluations)
batch_size: 86
dropout_1: 0.29046327939080685
dropout_2: 0.56812963618158
learning_rate: 0.00971849783517772
n_units_1: 128
n_units_2: 123
scale_1: 0.0010000000000000002
scale_2: 0.007699153336710522
Started BO from (top scorer):
batch_size: 84
dropout_1: 0.29045243854599945
dropout_2: 0.5681299155477846
learning_rate: 0.007410577045318821
n_units_1: 111
n_units_2: 123
scale_1: 0.11063448065875608
scale_2: 2.7172159701453054
Top score values: [-0.04052008  0.02178     0.0764601   0.1164286   0.12405922]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9, 15:1, 16:1, 16:3, 16:9, 17:1, 17:3, 17:9, 18:1, 18:3, 19:1, 19:3, 19:9, 20:1. Pending:
Targets: [ 0.10236402 -0.58102608 -0.87939783  1.85648914  2.13398716  2.18153558
  1.84294096  1.29246074  0.43876632 -0.08043309 -0.48842513  2.00273509
  0.66789798  0.31108543  1.50566202 -0.32304137 -0.65248902 -0.91268971
  0.23092441 -0.29543969 -0.07139749 -0.36192675  1.32495004 -0.50415817
 -0.92440615 -1.29546122 -0.64207802 -0.94597498 -1.11621356  1.20476581
 -0.50841963 -0.85100718 -1.08807495 -0.21457698 -0.70319436 -1.10910126
 -0.11244783 -0.35627533 -0.38953401 -0.73282503 -0.91642944 -0.04012042]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.24978167947420007, 'kernel_inv_bw1': 0.0008003164637168435, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 3.55902353920494, 'kernel_inv_bw4': 1.20011744943782, 'kernel_inv_bw5': 0.0010767226860239716, 'kernel_inv_bw6': 1.332453533412621, 'kernel_inv_bw7': 1.2456797098664971, 'kernel_inv_bw8': 0.7243831048153299, 'kernel_covariance_scale': 1.4911276462414562, 'mean_mean_value': 1.2398239116520988}
Epoch 1 ; Time: 0.302504 ; Training: accuracy=0.438562 ; Validation: accuracy=0.698339
Update for config_id 21:1: reward = 0.6983388704318937, crit_val = 0.3016611295681063
config_id 21: Reaches 1, continues to 3
Epoch 2 ; Time: 0.605059 ; Training: accuracy=0.631453 ; Validation: accuracy=0.780066
Epoch 3 ; Time: 0.892069 ; Training: accuracy=0.670790 ; Validation: accuracy=0.806146
Update for config_id 21:3: reward = 0.8061461794019934, crit_val = 0.1938538205980066
config_id 21: Reaches 3, continues to 9
Epoch 4 ; Time: 1.178961 ; Training: accuracy=0.699736 ; Validation: accuracy=0.822591
Epoch 5 ; Time: 1.490638 ; Training: accuracy=0.712024 ; Validation: accuracy=0.836545
Epoch 6 ; Time: 1.844467 ; Training: accuracy=0.724559 ; Validation: accuracy=0.847508
Epoch 7 ; Time: 2.177165 ; Training: accuracy=0.736104 ; Validation: accuracy=0.862458
Epoch 8 ; Time: 2.513269 ; Training: accuracy=0.743691 ; Validation: accuracy=0.856478
Epoch 9 ; Time: 2.850672 ; Training: accuracy=0.753670 ; Validation: accuracy=0.854983
config_id 21: Terminating evaluation at 9
Update for config_id 21:9: reward = 0.8549833887043189, crit_val = 0.1450166112956811
Starting get_config[BO] for config_id 22
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4288852711116653
- self.std = 0.2387063356373742
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 45
Current best is [0.13445378]
[22: BO] (31 evaluations)
batch_size: 8
dropout_1: 0.5597623370437255
dropout_2: 0.43944490965643446
learning_rate: 0.0005004578038988256
n_units_1: 70
n_units_2: 89
scale_1: 8.515949425434965
scale_2: 0.07218167085678881
Started BO from (top scorer):
batch_size: 71
dropout_1: 0.5597622717952918
dropout_2: 0.4394453462003949
learning_rate: 0.0009696499292183465
n_units_1: 97
n_units_2: 89
scale_1: 5.416358116563618
scale_2: 0.0010993055764807885
Top score values: [0.10849736 0.11287591 0.14163988 0.14519679 0.16528343]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9, 15:1, 16:1, 16:3, 16:9, 17:1, 17:3, 17:9, 18:1, 18:3, 19:1, 19:3, 19:9, 20:1, 21:1, 21:3, 21:9. Pending:
Targets: [ 0.16700323 -0.51766996 -0.81660192  1.92442178  2.20244082  2.25007851
  1.91084817  1.35933439  0.50403713 -0.01613709 -0.42489515  2.07094232
  0.733599    0.37611652  1.57293597 -0.25920088 -0.58926708 -0.8499563
  0.295805   -0.23154738 -0.00708453 -0.29815926  1.3918847  -0.44065773
 -0.86169474 -1.23344648 -0.57883653 -0.88330407 -1.05386228  1.27147482
 -0.44492719 -0.78815796 -1.02567084 -0.15053284 -0.64006762 -1.04673663
 -0.04821194 -0.29249724 -0.32581836 -0.66975392 -0.85370306  0.02425127
 -0.53297346 -0.984605   -1.18919617]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.24306654143612832, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00017957180150249163, 'kernel_inv_bw3': 3.6946631814676305, 'kernel_inv_bw4': 1.2534435597612723, 'kernel_inv_bw5': 0.0026664377144599394, 'kernel_inv_bw6': 1.2801691515722537, 'kernel_inv_bw7': 1.20290572647708, 'kernel_inv_bw8': 0.7528836236798467, 'kernel_covariance_scale': 1.393286444244258, 'mean_mean_value': 1.3453057287510302}
Epoch 1 ; Time: 2.893169 ; Training: accuracy=0.233173 ; Validation: accuracy=0.591184
Update for config_id 22:1: reward = 0.5911843876177658, crit_val = 0.4088156123822342
config_id 22: Reaches 1, continues to 3
Epoch 2 ; Time: 5.837823 ; Training: accuracy=0.372596 ; Validation: accuracy=0.640478
Epoch 3 ; Time: 8.764148 ; Training: accuracy=0.411555 ; Validation: accuracy=0.660162
config_id 22: Terminating evaluation at 3
Update for config_id 22:3: reward = 0.6601615074024226, crit_val = 0.33983849259757737
Starting get_config[BO] for config_id 23
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.42656364478733516
- self.std = 0.2339399337653385
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 47
Current best is [0.13445378]
[23: BO] (25 evaluations)
batch_size: 128
dropout_1: 0.3134983827249526
dropout_2: 0.7352587393338911
learning_rate: 0.0008338466269583976
n_units_1: 88
n_units_2: 51
scale_1: 0.7518272595398855
scale_2: 0.019313089787206573
Started BO from (top scorer):
batch_size: 85
dropout_1: 0.31349832499476277
dropout_2: 0.7352587949202476
learning_rate: 0.00033486684036548067
n_units_1: 97
n_units_2: 51
scale_1: 3.1394908510247905
scale_2: 0.2124395580502697
Top score values: [0.0300192  0.10144442 0.10614942 0.1195869  0.12029652]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9, 15:1, 16:1, 16:3, 16:9, 17:1, 17:3, 17:9, 18:1, 18:3, 19:1, 19:3, 19:9, 20:1, 21:1, 21:3, 21:9, 22:1, 22:3. Pending:
Targets: [ 0.18032986 -0.51829319 -0.82331572  1.97355488  2.25723841  2.30584669
  1.95970471  1.39695414  0.52423065 -0.00654185 -0.42362814  2.1230607
  0.75846972  0.39370372  1.61490773 -0.25455793 -0.59134905 -0.85734969
  0.3117559  -0.226341    0.00269515 -0.29431007  1.43016764 -0.43971187
 -0.86932729 -1.24865327 -0.58070599 -0.89137689 -1.06541013  1.30730447
 -0.44406832 -0.79429224 -1.03664431 -0.14367584 -0.64318463 -1.0581393
 -0.03927021 -0.28853269 -0.32253271 -0.67347577 -0.86117278  0.0346694
 -0.53390848 -0.99474177 -1.20350138 -0.07586577 -0.37071547]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.24680646738407225, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 3.812822596647508, 'kernel_inv_bw4': 1.2834061281882614, 'kernel_inv_bw5': 0.00029318259959548325, 'kernel_inv_bw6': 1.3105072234692994, 'kernel_inv_bw7': 1.1718830444446848, 'kernel_inv_bw8': 0.7597362042547287, 'kernel_covariance_scale': 1.3651439244674628, 'mean_mean_value': 1.371661215909066}
Epoch 1 ; Time: 0.198068 ; Training: accuracy=0.141941 ; Validation: accuracy=0.393617
config_id 23: Terminating evaluation at 1
Update for config_id 23:1: reward = 0.39361702127659576, crit_val = 0.6063829787234043

Analysing the results

The training history is stored in the results_df, the main fields are the runtime and 'best' (the objective).

Note: You will get slightly different curves for different pairs of scheduler/searcher, the time_out here is a bit too short to really see the difference in a significant way (it would be better to set it to >1000s). Generally speaking though, hyperband stopping / promotion + model will tend to significantly outperform other combinations given enough time.

results_df.head()
bracket elapsed_time epoch error eval_time objective runtime searcher_data_size searcher_params_kernel_covariance_scale searcher_params_kernel_inv_bw0 ... searcher_params_kernel_inv_bw7 searcher_params_kernel_inv_bw8 searcher_params_mean_mean_value searcher_params_noise_variance target_epoch task_id time_since_start time_step time_this_iter best
0 0 0.384673 1 0.468750 0.379464 0.531250 0.489095 NaN 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 0.492411 1.596679e+09 0.429917 0.468750
1 0 0.731945 2 0.344753 0.339371 0.655247 0.836367 1.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 0.837110 1.596679e+09 0.347255 0.344753
2 0 1.079010 3 0.305314 0.345247 0.694686 1.183432 1.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.184143 1.596679e+09 0.347064 0.305314
3 0 1.426687 4 0.288937 0.343776 0.711063 1.531108 2.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.531916 1.596679e+09 0.347676 0.288937
4 0 1.909700 5 0.273061 0.481130 0.726939 2.014122 2.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 2.014937 1.596679e+09 0.483014 0.273061

5 rows × 26 columns

import matplotlib.pyplot as plt

plt.figure(figsize=(12, 8))

runtime = results_df['runtime'].values
objective = results_df['best'].values

plt.plot(runtime, objective, lw=2)
plt.xticks(fontsize=12)
plt.xlim(0, 120)
plt.ylim(0, 0.5)
plt.yticks(fontsize=12)
plt.xlabel("Runtime [s]", fontsize=14)
plt.ylabel("Objective", fontsize=14)
Text(0, 0.5, 'Objective')
../../_images/output_mlp_cb387f_18_1.png

Diving Deeper

Now, you are ready to try HPO on your own machine learning models (if you use PyTorch, have a look at MNIST Training in PyTorch). While AutoGluon comes with well-chosen defaults, it can pay off to tune it to your specific needs. Here are some tips which may come useful.

Logging the Search Progress

First, it is a good idea in general to switch on debug_log, which outputs useful information about the search progress. This is already done in the example above.

The outputs show which configurations are chosen, stopped, or promoted. For BO and BOHB, a range of information is displayed for every get_config decision. This log output is very useful in order to figure out what is going on during the search.

Configuring HyperbandScheduler

The most important knobs to turn with HyperbandScheduler are max_t, grace_period, reduction_factor, brackets, and type. The first three determine the rung levels at which stopping or promotion decisions are being made.

  • The maximum resource level max_t (usually, resource equates to epochs, so max_t is the maximum number of training epochs) is typically hardcoded in train_fn passed to the scheduler (this is run_mlp_openml in the example above). As already noted above, the value is best fixed in the ag.args decorator as epochs=XYZ, it can then be accessed as args.epochs in the train_fn code. If this is done, you do not have to pass max_t when creating the scheduler.

  • grace_period and reduction_factor determine the rung levels, which are grace_period, grace_period * reduction_factor, grace_period * (reduction_factor ** 2), etc. All rung levels must be less or equal than max_t. It is recommended to make max_t equal to the largest rung level. For example, if grace_period = 1, reduction_factor = 3, it is in general recommended to use max_t = 9, max_t = 27, or max_t = 81. Choosing a max_t value “off the grid” works against the successive halving principle that the total resources spent in a rung should be roughly equal between rungs. If in the example above, you set max_t = 10, about a third of configurations reaching 9 epochs are allowed to proceed, but only for one more epoch.

  • With reduction_factor, you tune the extent to which successive halving filtering is applied. The larger this integer, the fewer configurations make it to higher number of epochs. Values 2, 3, 4 are commonly used.

  • Finally, grace_period should be set to the smallest resource (number of epochs) for which you expect any meaningful differentiation between configurations. While grace_period = 1 should always be explored, it may be too low for any meaningful stopping decisions to be made at the first rung.

  • brackets sets the maximum number of brackets in Hyperband (make sure to study the Hyperband paper or follow-ups for details). For brackets = 1, you are running successive halving (single bracket). Higher brackets have larger effective grace_period values (so runs are not stopped until later), yet are also chosen with less probability. We recommend to always consider successive halving (brackets = 1) in a comparison.

  • Finally, with type (values stopping, promotion) you are choosing different ways of extending successive halving scheduling to the asynchronous case. The method for the default stopping is simpler and seems to perform well, but promotion is more careful promoting configurations to higher resource levels, which can work better in some cases.

Asynchronous BOHB

Finally, here are some ideas for tuning asynchronous BOHB, apart from tuning its HyperbandScheduling component. You need to pass these options in search_options.

  • We support a range of different surrogate models over the criterion functions across resource levels. All of them are jointly dependent Gaussian process models, meaning that data collected at all resource levels are modelled together. The surrogate model is selected by gp_resource_kernel, values are matern52, matern52-res-warp, exp-decay-sum, exp-decay-combined, exp-decay-delta1. These are variants of either a joint Matern 5/2 kernel over configuration and resource, or the exponential decay model. Details about the latter can be found here.

  • Fitting a Gaussian process surrogate model to data encurs a cost which scales cubically with the number of datapoints. When applied to expensive deep learning workloads, even multi-fidelity asynchronous BOHB is rarely running up more than 100 observations or so (across all rung levels and brackets), and the GP computations are subdominant. However, if you apply it to cheaper train_fn and find yourself beyond 2000 total evaluations, the cost of GP fitting can become painful. In such a situation, you can explore the options opt_skip_period and opt_skip_num_max_resource. The basic idea is as follows. By far the most expensive part of a get_config call (picking the next configuration) is the refitting of the GP model to past data (this entails re-optimizing hyperparameters of the surrogate model itself). The options allow you to skip this expensive step for most get_config calls, after some initial period. Check the docstrings for details about these options. If you find yourself in such a situation and gain experience with these skipping features, make sure to contact the AutoGluon developers – we would love to learn about your use case.