Predicting Columns in a Table - In Depth

Tip: If you are new to AutoGluon, review Predicting Columns in a Table - Quick Start to learn the basics of the AutoGluon API.

This tutorial describes how you can exert greater control when using AutoGluon’s fit() by specifying the appropriate arguments. Using the same census data table as Predicting Columns in a Table - Quick Start, we will try to predict the occupation of an individual - a multi-class classification problem.

Start by importing AutoGluon, specifying TabularPrediction as the task, and loading the data.

import autogluon as ag
from autogluon import TabularPrediction as task

train_data = task.Dataset(file_path='https://autogluon.s3.amazonaws.com/datasets/Inc/train.csv')
train_data = train_data.head(500) # subsample 500 data points for faster demo
print(train_data.head())

val_data = task.Dataset(file_path='https://autogluon.s3.amazonaws.com/datasets/Inc/test.csv')

label_column = 'occupation'
print("Summary of occupation column: \n", train_data['occupation'].describe())
/var/lib/jenkins/miniconda3/envs/autogluon_docs/lib/python3.7/site-packages/sklearn/utils/deprecation.py:144: FutureWarning: The sklearn.metrics.classification module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.metrics. Anything that cannot be imported from sklearn.metrics is now part of the private API.
  warnings.warn(message, FutureWarning)
Loaded data from: https://autogluon.s3.amazonaws.com/datasets/Inc/train.csv | Columns = 15 / 15 | Rows = 39073 -> 39073
Loaded data from: https://autogluon.s3.amazonaws.com/datasets/Inc/test.csv | Columns = 15 / 15 | Rows = 9769 -> 9769
   age   workclass  fnlwgt   education  education-num       marital-status  0   25     Private  178478   Bachelors             13        Never-married
1   23   State-gov   61743     5th-6th              3        Never-married
2   46     Private  376789     HS-grad              9        Never-married
3   55           ?  200235     HS-grad              9   Married-civ-spouse
4   36     Private  224541     7th-8th              4   Married-civ-spouse

           occupation    relationship    race      sex  capital-gain  0        Tech-support       Own-child   White   Female             0
1    Transport-moving   Not-in-family   White     Male             0
2       Other-service   Not-in-family   White     Male             0
3                   ?         Husband   White     Male             0
4   Handlers-cleaners         Husband   White     Male             0

   capital-loss  hours-per-week  native-country   class
0             0              40   United-States   <=50K
1             0              35   United-States   <=50K
2             0              15   United-States   <=50K
3             0              50   United-States    >50K
4             0              40     El-Salvador   <=50K
Summary of occupation column:
 count                  500
unique                  14
top        Exec-managerial
freq                    69
Name: occupation, dtype: object

To demonstrate how you can provide your own validation dataset against which AutoGluon tunes hyperparameters, we’ll use the test dataset from the previous tutorial as validation data.

If you do not wish to provide a validation dataset, it is recommended that you omit the tuning_data argument. This lets AutoGluon automatically select validation data from your provided training set (it uses smart strategies such as stratified sampling). For greater control, you can specify the holdout_frac argument to tell AutoGluon what fraction of the provided training data to hold out for validation.

Caution: Since AutoGluon tunes internal knobs based on this validation data, performance estimates reported on this data may be over-optimistic. For unbiased performance estimates, you should always call predict() on a separate dataset (that was never passed to fit()), as we did in the previous Quick-Start tutorial. We also emphasize that most options specified in this tutorial are chosen to minimize runtime for the purposes of demonstration and you should select more reasonable values in order to obtain high-quality models.

fit() trains neural networks and various types of tree ensembles by default. You can specify various hyperparameter values for each type of model. For each hyperparameter, you can either specify a single fixed value, or a search space of values to consider during the hyperparameter optimization. Hyperparameters which you do not specify are left at default settings chosen by AutoGluon, which may be fixed values or search spaces, depending on the particular hyperparameter and the setting of hyperparameter_tune.

hp_tune = True  # whether or not to do hyperparameter optimization

nn_options = { # specifies non-default hyperparameter values for neural network models
    'num_epochs': 10, # number of training epochs (controls training time of NN models)
    'learning_rate': ag.space.Real(1e-4, 1e-2, default=5e-4, log=True), # learning rate used in training (real-valued hyperparameter searched on log-scale)
    'activation': ag.space.Categorical('relu', 'softrelu', 'tanh'), # activation function used in NN (categorical hyperparameter, default = first entry)
    'layers': ag.space.Categorical([100],[1000],[200,100],[300,200,100]),
      # Each choice for categorical hyperparameter 'layers' corresponds to list of sizes for each NN layer to use
    'dropout_prob': ag.space.Real(0.0, 0.5, default=0.1), # dropout probability (real-valued hyperparameter)
}

gbm_options = { # specifies non-default hyperparameter values for lightGBM gradient boosted trees
    'num_boost_round': 100, # number of boosting rounds (controls training time of GBM models)
    'num_leaves': ag.space.Int(lower=26, upper=66, default=36), # number of leaves in trees (integer hyperparameter)
}

hyperparameters = {'NN': nn_options, 'GBM': gbm_options}  # hyperparameters of each model type
# If one of these keys is missing from hyperparameters dict, then no models of that type are trained.

time_limits = 2*60  # train various models for ~2 min
num_trials = 5  # try at most 3 different hyperparameter configurations for each type of model
search_strategy = 'skopt'  # to tune hyperparameters using SKopt Bayesian optimization routine
output_directory = 'agModels-predictOccupation'  # folder where to store trained models

predictor = task.fit(train_data=train_data, tuning_data=val_data, label=label_column,
                     output_directory=output_directory, time_limits=time_limits, num_trials=num_trials,
                     hyperparameter_tune=hp_tune, hyperparameters=hyperparameters,
                     search_strategy=search_strategy)
Warning: hyperparameter_tune=True is currently experimental and may cause the process to hang. Setting auto_stack=True instead is recommended to achieve maximum quality models.
Beginning AutoGluon training ... Time limit = 120s
AutoGluon will save models to agModels-predictOccupation/
Train Data Rows:    500
Train Data Columns: 15
Tuning Data Rows:    9769
Tuning Data Columns: 15
Preprocessing data ...
Here are the first 10 unique label values in your data:  [' Tech-support' ' Transport-moving' ' Other-service' ' ?'
 ' Handlers-cleaners' ' Sales' ' Craft-repair' ' Adm-clerical'
 ' Exec-managerial' ' Prof-specialty']
AutoGluon infers your prediction problem is: multiclass  (because dtype of label-column == object)
If this is wrong, please specify problem_type argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])

Warning: Some classes in the training set have fewer than 10 examples. AutoGluon will only keep 13 out of 14 classes for training and will not try to predict the rare classes. To keep more classes, increase the number of datapoints from these rare classes in the training data or reduce label_count_threshold.
Fraction of data from classes with at least 10 examples that will be kept for training models: 0.998
Feature Generator processed 10223 data points with 14 features
Original Features:
    int features: 6
    object features: 8
Generated Features:
    int features: 0
All Features:
    int features: 6
    object features: 8
    Data preprocessing and feature engineering runtime = 0.09s ...
AutoGluon will gauge predictive performance using evaluation metric: accuracy
To change this, specify the eval_metric argument of fit()
AutoGluon will early stop models using evaluation metric: accuracy
Starting Experiments
Num of Finished Tasks is 0
Num of Pending Tasks is 5
HBox(children=(FloatProgress(value=0.0, max=5.0), HTML(value='')))
Finished Task with config: {'feature_fraction': 1.0, 'learning_rate': 0.1, 'min_data_in_leaf': 20, 'num_leaves': 36} and reward: 0.28661044837515426
Finished Task with config: b'x80x03}qx00(Xx10x00x00x00feature_fractionqx01G?xf0x00x00x00x00x00x00Xrx00x00x00learning_rateqx02G?xb9x99x99x99x99x99x9aXx10x00x00x00min_data_in_leafqx03Kx14Xnx00x00x00num_leavesqx04K$u.' and reward: 0.28661044837515426
Finished Task with config: b'x80x03}qx00(Xx10x00x00x00feature_fractionqx01G?xf0x00x00x00x00x00x00Xrx00x00x00learning_rateqx02G?xb9x99x99x99x99x99x9aXx10x00x00x00min_data_in_leafqx03Kx14Xnx00x00x00num_leavesqx04K$u.' and reward: 0.28661044837515426
Finished Task with config: {'feature_fraction': 0.9172982732529726, 'learning_rate': 0.04024904647521891, 'min_data_in_leaf': 19, 'num_leaves': 35} and reward: 0.28691896338955164
Finished Task with config: b'x80x03}qx00(Xx10x00x00x00feature_fractionqx01G?xedZx81xe8x89x8fxaeXrx00x00x00learning_rateqx02G?xa4x9bx85xe4x9cxccxb5Xx10x00x00x00min_data_in_leafqx03Kx13Xnx00x00x00num_leavesqx04K#u.' and reward: 0.28691896338955164
Finished Task with config: b'x80x03}qx00(Xx10x00x00x00feature_fractionqx01G?xedZx81xe8x89x8fxaeXrx00x00x00learning_rateqx02G?xa4x9bx85xe4x9cxccxb5Xx10x00x00x00min_data_in_leafqx03Kx13Xnx00x00x00num_leavesqx04K#u.' and reward: 0.28691896338955164
Finished Task with config: {'feature_fraction': 0.8905885993654692, 'learning_rate': 0.008548987190436536, 'min_data_in_leaf': 9, 'num_leaves': 44} and reward: 0.27735499794323326
Finished Task with config: b'x80x03}qx00(Xx10x00x00x00feature_fractionqx01G?xecx7fxb3xa9x8exe2fXrx00x00x00learning_rateqx02G?x81x82!xa3,xbex90Xx10x00x00x00min_data_in_leafqx03KtXnx00x00x00num_leavesqx04K,u.' and reward: 0.27735499794323326
Finished Task with config: b'x80x03}qx00(Xx10x00x00x00feature_fractionqx01G?xecx7fxb3xa9x8exe2fXrx00x00x00learning_rateqx02G?x81x82!xa3,xbex90Xx10x00x00x00min_data_in_leafqx03KtXnx00x00x00num_leavesqx04K,u.' and reward: 0.27735499794323326
Finished Task with config: {'feature_fraction': 0.9739603794195677, 'learning_rate': 0.02218282034173824, 'min_data_in_leaf': 9, 'num_leaves': 60} and reward: 0.27457836281365694
Finished Task with config: b'x80x03}qx00(Xx10x00x00x00feature_fractionqx01G?xef*xaexf5&x9ex10Xrx00x00x00learning_rateqx02G?x96xb7x17xdfx9amxc0Xx10x00x00x00min_data_in_leafqx03KtXnx00x00x00num_leavesqx04K<u.' and reward: 0.27457836281365694
Finished Task with config: b'x80x03}qx00(Xx10x00x00x00feature_fractionqx01G?xef*xaexf5&x9ex10Xrx00x00x00learning_rateqx02G?x96xb7x17xdfx9amxc0Xx10x00x00x00min_data_in_leafqx03KtXnx00x00x00num_leavesqx04K<u.' and reward: 0.27457836281365694
Finished Task with config: {'feature_fraction': 0.8492569278852602, 'learning_rate': 0.025772989470668584, 'min_data_in_leaf': 7, 'num_leaves': 63} and reward: 0.27478403948992186
Finished Task with config: b'x80x03}qx00(Xx10x00x00x00feature_fractionqx01G?xeb-x1cxddee\Xrx00x00x00learning_rateqx02G?x9ad<x0bx961xb0Xx10x00x00x00min_data_in_leafqx03Kx07Xnx00x00x00num_leavesqx04K?u.' and reward: 0.27478403948992186
Finished Task with config: b'x80x03}qx00(Xx10x00x00x00feature_fractionqx01G?xeb-x1cxddee\Xrx00x00x00learning_rateqx02G?x9ad<x0bx961xb0Xx10x00x00x00min_data_in_leafqx03Kx07Xnx00x00x00num_leavesqx04K?u.' and reward: 0.27478403948992186
Starting Experiments
Num of Finished Tasks is 0
Num of Pending Tasks is 5
HBox(children=(FloatProgress(value=0.0, max=5.0), HTML(value='')))
Finished Task with config: {'activation.choice': 0, 'dropout_prob': 0.1, 'embedding_size_factor': 1.0, 'layers.choice': 0, 'learning_rate': 0.0005, 'network_type.choice': 0, 'use_batchnorm.choice': 0, 'weight_decay': 1e-06} and reward: 0.12659399424105305
Finished Task with config: b'x80x03}qx00(Xx11x00x00x00activation.choiceqx01Kx00Xx0cx00x00x00dropout_probqx02G?xb9x99x99x99x99x99x9aXx15x00x00x00embedding_size_factorqx03G?xf0x00x00x00x00x00x00Xrx00x00x00layers.choiceqx04Kx00Xrx00x00x00learning_rateqx05G?@bMxd2xf1xa9xfcXx13x00x00x00network_type.choiceqx06Kx00Xx14x00x00x00use_batchnorm.choiceqx07Kx00Xx0cx00x00x00weight_decayqx08G>xb0xc6xf7xa0xb5xedx8du.' and reward: 0.12659399424105305
Finished Task with config: b'x80x03}qx00(Xx11x00x00x00activation.choiceqx01Kx00Xx0cx00x00x00dropout_probqx02G?xb9x99x99x99x99x99x9aXx15x00x00x00embedding_size_factorqx03G?xf0x00x00x00x00x00x00Xrx00x00x00layers.choiceqx04Kx00Xrx00x00x00learning_rateqx05G?@bMxd2xf1xa9xfcXx13x00x00x00network_type.choiceqx06Kx00Xx14x00x00x00use_batchnorm.choiceqx07Kx00Xx0cx00x00x00weight_decayqx08G>xb0xc6xf7xa0xb5xedx8du.' and reward: 0.12659399424105305
Finished Task with config: {'activation.choice': 0, 'dropout_prob': 0.24875708993596046, 'embedding_size_factor': 1.3975402192971433, 'layers.choice': 1, 'learning_rate': 0.00012000490407837813, 'network_type.choice': 0, 'use_batchnorm.choice': 1, 'weight_decay': 3.559024856901967e-08} and reward: 0.16670094611271083
Finished Task with config: b'x80x03}qx00(Xx11x00x00x00activation.choiceqx01Kx00Xx0cx00x00x00dropout_probqx02G?xcfxd7Exb6xf6'x84Xx15x00x00x00embedding_size_factorqx03G?xf6\S"x0bx9dHXrx00x00x00layers.choiceqx04Kx01Xrx00x00x00learning_rateqx05G?x1fudx8dxb2x81x04Xx13x00x00x00network_type.choiceqx06Kx00Xx14x00x00x00use_batchnorm.choiceqx07Kx01Xx0cx00x00x00weight_decayqx08G>cx1b|x8ccr;u.' and reward: 0.16670094611271083
Finished Task with config: b'x80x03}qx00(Xx11x00x00x00activation.choiceqx01Kx00Xx0cx00x00x00dropout_probqx02G?xcfxd7Exb6xf6'x84Xx15x00x00x00embedding_size_factorqx03G?xf6\S"x0bx9dHXrx00x00x00layers.choiceqx04Kx01Xrx00x00x00learning_rateqx05G?x1fudx8dxb2x81x04Xx13x00x00x00network_type.choiceqx06Kx00Xx14x00x00x00use_batchnorm.choiceqx07Kx01Xx0cx00x00x00weight_decayqx08G>cx1b|x8ccr;u.' and reward: 0.16670094611271083
Finished Task with config: {'activation.choice': 0, 'dropout_prob': 0.32522156555011167, 'embedding_size_factor': 1.041332796106567, 'layers.choice': 0, 'learning_rate': 0.002828290699886094, 'network_type.choice': 1, 'use_batchnorm.choice': 0, 'weight_decay': 2.267863119770332e-10} and reward: 0.23755656108597284
Finished Task with config: b'x80x03}qx00(Xx11x00x00x00activation.choiceqx01Kx00Xx0cx00x00x00dropout_probqx02G?xd4xd0nx1cxffw1Xx15x00x00x00embedding_size_factorqx03G?xf0xa9Lx93xf8zxa4Xrx00x00x00layers.choiceqx04Kx00Xrx00x00x00learning_rateqx05G?g+[x01xe8,xe8Xx13x00x00x00network_type.choiceqx06Kx01Xx14x00x00x00use_batchnorm.choiceqx07Kx00Xx0cx00x00x00weight_decayqx08G=xef+Ux80x0exbexbbu.' and reward: 0.23755656108597284
Finished Task with config: b'x80x03}qx00(Xx11x00x00x00activation.choiceqx01Kx00Xx0cx00x00x00dropout_probqx02G?xd4xd0nx1cxffw1Xx15x00x00x00embedding_size_factorqx03G?xf0xa9Lx93xf8zxa4Xrx00x00x00layers.choiceqx04Kx00Xrx00x00x00learning_rateqx05G?g+[x01xe8,xe8Xx13x00x00x00network_type.choiceqx06Kx01Xx14x00x00x00use_batchnorm.choiceqx07Kx00Xx0cx00x00x00weight_decayqx08G=xef+Ux80x0exbexbbu.' and reward: 0.23755656108597284
Finished Task with config: {'activation.choice': 0, 'dropout_prob': 0.4619864251770406, 'embedding_size_factor': 0.9282821396898141, 'layers.choice': 0, 'learning_rate': 0.0006840054269763689, 'network_type.choice': 0, 'use_batchnorm.choice': 0, 'weight_decay': 1.507872416598938e-05} and reward: 0.16217605923488276
Finished Task with config: b'x80x03}qx00(Xx11x00x00x00activation.choiceqx01Kx00Xx0cx00x00x00dropout_probqx02G?xddx91/x82xd54xaeXx15x00x00x00embedding_size_factorqx03G?xedxb4|xbexedxb7x8bXrx00x00x00layers.choiceqx04Kx00Xrx00x00x00learning_rateqx05G?Fixdax6x11`Xx13x00x00x00network_type.choiceqx06Kx00Xx14x00x00x00use_batchnorm.choiceqx07Kx00Xx0cx00x00x00weight_decayqx08G>xefx9fTx11xaeX3u.' and reward: 0.16217605923488276
Finished Task with config: b'x80x03}qx00(Xx11x00x00x00activation.choiceqx01Kx00Xx0cx00x00x00dropout_probqx02G?xddx91/x82xd54xaeXx15x00x00x00embedding_size_factorqx03G?xedxb4|xbexedxb7x8bXrx00x00x00layers.choiceqx04Kx00Xrx00x00x00learning_rateqx05G?Fixdax6x11`Xx13x00x00x00network_type.choiceqx06Kx00Xx14x00x00x00use_batchnorm.choiceqx07Kx00Xx0cx00x00x00weight_decayqx08G>xefx9fTx11xaeX3u.' and reward: 0.16217605923488276
Finished Task with config: {'activation.choice': 2, 'dropout_prob': 0.19175063855671765, 'embedding_size_factor': 1.1060897339100555, 'layers.choice': 3, 'learning_rate': 0.00024377745284882347, 'network_type.choice': 0, 'use_batchnorm.choice': 0, 'weight_decay': 5.702137177684393e-07} and reward: 0.24259563965446318
Finished Task with config: b'x80x03}qx00(Xx11x00x00x00activation.choiceqx01Kx02Xx0cx00x00x00dropout_probqx02G?xc8x8bHxf0xcbJxc2Xx15x00x00x00embedding_size_factorqx03G?xf1xb2x8b&x19\IXrx00x00x00layers.choiceqx04Kx03Xrx00x00x00learning_rateqx05G?/xf3xd0_xfeiHXx13x00x00x00network_type.choiceqx06Kx00Xx14x00x00x00use_batchnorm.choiceqx07Kx00Xx0cx00x00x00weight_decayqx08G>xa3"x199xdbxa1xb2u.' and reward: 0.24259563965446318
Finished Task with config: b'x80x03}qx00(Xx11x00x00x00activation.choiceqx01Kx02Xx0cx00x00x00dropout_probqx02G?xc8x8bHxf0xcbJxc2Xx15x00x00x00embedding_size_factorqx03G?xf1xb2x8b&x19\IXrx00x00x00layers.choiceqx04Kx03Xrx00x00x00learning_rateqx05G?/xf3xd0_xfeiHXx13x00x00x00network_type.choiceqx06Kx00Xx14x00x00x00use_batchnorm.choiceqx07Kx00Xx0cx00x00x00weight_decayqx08G>xa3"x199xdbxa1xb2u.' and reward: 0.24259563965446318
Please either provide filename or allow plot in get_training_curves
Fitting model: weighted_ensemble_k0_l1 ... Training model for up to 119.91s of the 31.51s of remaining time.
    0.3016   = Validation accuracy score
    2.78s    = Training runtime
    0.0s     = Validation runtime
AutoGluon training complete, total runtime = 91.29s ...
../../_images/output_tabular-indepth_108df8_3_9.png

We again demonstrate how to use the trained models to predict on the validation data. We caution again that performance estimates from this data may be biased because the same data was used to tune hyperparameters.

test_data = val_data.copy()
y_test = test_data[label_column]
test_data = test_data.drop(labels=[label_column],axis=1)  # delete label column

y_pred = predictor.predict(test_data)
print("Predictions:  ", list(y_pred)[:5])
perf = predictor.evaluate_predictions(y_true=y_test, y_pred=y_pred, auxiliary_metrics=False)
Evaluation: accuracy on test data: 0.300235
Predictions:   [' Other-service', ' ?', ' Exec-managerial', ' Sales', ' Other-service']

predictor can also make a prediction on an individual example rather than a full dataset:

datapoint = test_data.iloc[[0]]  # Note: .iloc[0] won't work because it returns pandas Series instead of DataFrame
print(datapoint)
print("Prediction:", predictor.predict(datapoint))
   age workclass  fnlwgt education  education-num       marital-status  0   31   Private  169085      11th              7   Married-civ-spouse

  relationship    race      sex  capital-gain  capital-loss  hours-per-week  0         Wife   White   Female             0             0              20

   native-country   class
0   United-States   <=50K
Prediction: [' Other-service']

Use the following to view a summary of what happened during fit. This command will shows details of the hyperparameter-tuning process for each type of model:

results = predictor.fit_summary()
* Summary of fit() *
Number of models trained: 11
Types of models trained:
{'TabularNeuralNetModel', 'LGBModel', 'WeightedEnsembleModel'}
Validation performance of individual models: {'LightGBMClassifier_trial_0': 0.28661044837515426, 'LightGBMClassifier_trial_1': 0.28691896338955164, 'LightGBMClassifier_trial_2': 0.27735499794323326, 'LightGBMClassifier_trial_3': 0.27457836281365694, 'LightGBMClassifier_trial_4': 0.27478403948992186, 'NeuralNetClassifier_trial_5': 0.12659399424105305, 'NeuralNetClassifier_trial_6': 0.16670094611271083, 'NeuralNetClassifier_trial_7': 0.23755656108597284, 'NeuralNetClassifier_trial_8': 0.16217605923488276, 'NeuralNetClassifier_trial_9': 0.24259563965446318, 'weighted_ensemble_k0_l1': 0.3016248457424928}
Best model (based on validation performance): weighted_ensemble_k0_l1
Hyperparameter-tuning used: True
Bagging used: False
Stack-ensembling used: False
User-specified hyperparameters:
{'NN': {'num_epochs': 10, 'learning_rate': Real: lower=0.0001, upper=0.01, 'activation': Categorical['relu', 'softrelu', 'tanh'], 'layers': Categorical[[100], [1000], [200, 100], [300, 200, 100]], 'dropout_prob': Real: lower=0.0, upper=0.5}, 'GBM': {'num_boost_round': 100, 'num_leaves': Int: lower=26, upper=66}}
Plot summary of models saved to file: SummaryOfModels.html
Plot summary of models saved to file: LightGBMClassifier_HPOmodelsummary.html
Plot of HPO performance saved to file: agModels-predictOccupation/LightGBMClassifier_HPOperformanceVStrials.png
../../_images/output_tabular-indepth_108df8_9_1.png
Plot summary of models saved to file: NeuralNetClassifier_HPOmodelsummary.html
Plot of HPO performance saved to file: agModels-predictOccupation/NeuralNetClassifier_HPOperformanceVStrials.png
../../_images/output_tabular-indepth_108df8_9_3.png
* Details of Hyperparameter optimization *
HPO for LightGBMClassifier model:  Num. configurations tried = 5, Time spent = 52.424522399902344, Search strategy = skopt
Best hyperparameter-configuration (validation-performance: accuracy = 0.28691896338955164):
{'feature_fraction': 0.9172982732529726, 'learning_rate': 0.04024904647521891, 'min_data_in_leaf': 19, 'num_leaves': 35}
HPO for NeuralNetClassifier model:  Num. configurations tried = 5, Time spent = 32.05798697471619, Search strategy = skopt
Best hyperparameter-configuration (validation-performance: accuracy = 0.24259563965446318):
{'activation.choice': 2, 'dropout_prob': 0.19175063855671765, 'embedding_size_factor': 1.1060897339100555, 'layers.choice': 3, 'learning_rate': 0.00024377745284882347, 'network_type.choice': 0, 'use_batchnorm.choice': 0, 'weight_decay': 5.702137177684393e-07}
* End of fit() summary *

In the above example, the predictive performance may be poor because we specified very little training to ensure quick runtimes. You can call fit() multiple times while modifying the above settings to better understand how these choices affect performance outcomes. For example, start by commenting out the train_data.head command to train using a larger dataset. To see more detailed output during the execution of fit(), you can also pass in the argument: verbosity = 3.

Performance in certain applications may be measured by different metrics than the ones AutoGluon optimizes for by default. If you know the metric that counts most in your application, you can specify it as done below:

metric = 'balanced_accuracy' # Use balanced accuracy rather than standard accuracy. You can also define your own function here, see examples in: autogluon/utils/tabular/metrics/
predictor = task.fit(train_data=train_data, label=label_column, eval_metric=metric)

performance = predictor.evaluate(val_data)
No output_directory specified. Models will be saved in: AutogluonModels/ag-20200218_015556/
Beginning AutoGluon training ...
AutoGluon will save models to AutogluonModels/ag-20200218_015556/
Train Data Rows:    500
Train Data Columns: 15
Preprocessing data ...
Here are the first 10 unique label values in your data:  [' Tech-support' ' Transport-moving' ' Other-service' ' ?'
 ' Handlers-cleaners' ' Sales' ' Craft-repair' ' Adm-clerical'
 ' Exec-managerial' ' Prof-specialty']
AutoGluon infers your prediction problem is: multiclass  (because dtype of label-column == object)
If this is wrong, please specify problem_type argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])

Warning: Some classes in the training set have fewer than 10 examples. AutoGluon will only keep 13 out of 14 classes for training and will not try to predict the rare classes. To keep more classes, increase the number of datapoints from these rare classes in the training data or reduce label_count_threshold.
Fraction of data from classes with at least 10 examples that will be kept for training models: 0.998
Feature Generator processed 499 data points with 14 features
Original Features:
    int features: 6
    object features: 8
Generated Features:
    int features: 0
All Features:
    int features: 6
    object features: 8
    Data preprocessing and feature engineering runtime = 0.05s ...
AutoGluon will gauge predictive performance using evaluation metric: balanced_accuracy
To change this, specify the eval_metric argument of fit()
AutoGluon will early stop models using evaluation metric: balanced_accuracy
/var/lib/jenkins/miniconda3/envs/autogluon_docs/lib/python3.7/imp.py:342: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  return _load(spec)
Fitting model: RandomForestClassifierGini ...
    0.1981   = Validation balanced_accuracy score
    0.61s    = Training runtime
    0.11s    = Validation runtime
Fitting model: RandomForestClassifierEntr ...
    0.1801   = Validation balanced_accuracy score
    0.61s    = Training runtime
    0.11s    = Validation runtime
Fitting model: ExtraTreesClassifierGini ...
    0.2145   = Validation balanced_accuracy score
    0.5s     = Training runtime
    0.11s    = Validation runtime
Fitting model: ExtraTreesClassifierEntr ...
    0.1821   = Validation balanced_accuracy score
    0.5s     = Training runtime
    0.11s    = Validation runtime
Fitting model: KNeighborsClassifierUnif ...
    0.0576   = Validation balanced_accuracy score
    0.01s    = Training runtime
    0.11s    = Validation runtime
Fitting model: KNeighborsClassifierDist ...
    0.0692   = Validation balanced_accuracy score
    0.0s     = Training runtime
    0.11s    = Validation runtime
Fitting model: LightGBMClassifier ...
    0.157    = Validation balanced_accuracy score
    7.29s    = Training runtime
    0.01s    = Validation runtime
Fitting model: CatboostClassifier ...
    0.2182   = Validation balanced_accuracy score
    4.95s    = Training runtime
    0.01s    = Validation runtime
Fitting model: NeuralNetClassifier ...
    0.1835   = Validation balanced_accuracy score
    4.5s     = Training runtime
    0.12s    = Validation runtime
Fitting model: LightGBMClassifierCustom ...
    0.1795   = Validation balanced_accuracy score
    58.27s   = Training runtime
    0.01s    = Validation runtime
Fitting model: weighted_ensemble_k0_l1 ...
    0.2497   = Validation balanced_accuracy score
    0.46s    = Training runtime
    0.0s     = Validation runtime
AutoGluon training complete, total runtime = 80.1s ...
Predictive performance on given dataset: balanced_accuracy = 0.25150402633613217

Beyond hyperparameter-tuning with a correctly-specified metric, two other methods to boost predictive performance are bagging and stack-ensembling. You’ll often see performance improve if you specify num_bagging_folds = 5-10, stack_ensemble_levels = 1 or 2 in the call to fit(), but this will increase training times.

predictor = task.fit(train_data=train_data, label=label_column, eval_metric=metric, num_bagging_folds=5, stack_ensemble_levels=1, hyperparameters = {'NN':{'num_epochs':5}, 'GBM':{'num_boost_round':100}})
No output_directory specified. Models will be saved in: AutogluonModels/ag-20200218_015717/
Beginning AutoGluon training ...
AutoGluon will save models to AutogluonModels/ag-20200218_015717/
Train Data Rows:    500
Train Data Columns: 15
Preprocessing data ...
Here are the first 10 unique label values in your data:  [' Tech-support' ' Transport-moving' ' Other-service' ' ?'
 ' Handlers-cleaners' ' Sales' ' Craft-repair' ' Adm-clerical'
 ' Exec-managerial' ' Prof-specialty']
AutoGluon infers your prediction problem is: multiclass  (because dtype of label-column == object)
If this is wrong, please specify problem_type argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])

Warning: Some classes in the training set have fewer than 10 examples. AutoGluon will only keep 13 out of 14 classes for training and will not try to predict the rare classes. To keep more classes, increase the number of datapoints from these rare classes in the training data or reduce label_count_threshold.
Fraction of data from classes with at least 10 examples that will be kept for training models: 0.998
Feature Generator processed 499 data points with 14 features
Original Features:
    int features: 6
    object features: 8
Generated Features:
    int features: 0
All Features:
    int features: 6
    object features: 8
    Data preprocessing and feature engineering runtime = 0.05s ...
AutoGluon will gauge predictive performance using evaluation metric: balanced_accuracy
To change this, specify the eval_metric argument of fit()
AutoGluon will early stop models using evaluation metric: balanced_accuracy
Fitting model: LightGBMClassifier_STACKER_l0 ...
    0.2122   = Validation balanced_accuracy score
    22.28s   = Training runtime
    0.0s     = Validation runtime
Fitting model: NeuralNetClassifier_STACKER_l0 ...
    0.0923   = Validation balanced_accuracy score
    5.73s    = Training runtime
    0.0s     = Validation runtime
Fitting model: weighted_ensemble_k0_l1 ...
    0.2122   = Validation balanced_accuracy score
    0.21s    = Training runtime
    0.0s     = Validation runtime
Fitting model: LightGBMClassifier_STACKER_l1 ...
    0.2169   = Validation balanced_accuracy score
    23.69s   = Training runtime
    0.0s     = Validation runtime
Fitting model: NeuralNetClassifier_STACKER_l1 ...
    0.0974   = Validation balanced_accuracy score
    6.12s    = Training runtime
    0.0s     = Validation runtime
Fitting model: weighted_ensemble_k0_l2 ...
    0.2169   = Validation balanced_accuracy score
    0.21s    = Training runtime
    0.0s     = Validation runtime
AutoGluon training complete, total runtime = 58.34s ...