Hyperparameter tuning using Bayesian optimization

I am using CNN for a binary classification problem and need to use Bayesian optimization to tune parameters like learning rate, number of hidden layers, optimizers, etc

I have built my model with fixed learning rate and number of epochs and Adam optimizer. Can you please give me some guidance on where to go from here in order to have Bayesian optimization implemented for tuning these parameters? And how would I deal with my 3 data sets (training, validation and test)?

I will really appreciate your help.

1 Like

You don’t need to do anything special to perform bayesian optimization for your hyperparameter tuning when using pytorch. You could just setup a script with command line arguments like --learning_rate, --num_layers for the hyperparameters you want to tune and maybe have a second script that calls this script with the diff. hyperparameter values in your bayesian parameter optimization loop.

Conceptually, you can do sth like this

for round i in max_rounds:
     output = subprocess.call('model.py', '--learning_rate', 'some_val', '--num_layers',  'some_number')
     score = # parse score from output string (e.g., your loss output)
     #optimize p(score | hyperparam values)
     # get some_val, some_number for next round

But it is probably more efficient if you use a library like hyperopt (https://github.com/hyperopt/hyperopt)

Maybe sth like

def run_model(learning_rate):
    # your model init here
    # your training here
    return loss

import numpy as np
from hyperopt import hp, tpe, fmin

# Single line bayesian optimization of polynomial function
best = fmin(fn=lambda x: run_model(x),
            space=hp.normal('x', 0.00001, 0.9),

But given how expensive typical model training for DL is, it may take a long time …


I went through similar problem only mine was hyperparameter tuning for feedforward neural networks. Then I ran into this very helpful article, you should check it:

Looks like this link is down :frowning:

Yup the link is down FYI @eph_adm

Hi All,

I want to do hyper parameter tuning for CNN layers ( 2 or 3 layers), number of filters for CNN, FC layers ( 2 or 3 layers) and number of neurons ([100:10:100]) , batch size {100,200}, LR {10^-4,10^-5}, Dropout{0.3,0.5,0.7}.

Do you know any Pytorch package to implement as grid search?

you can use skorch for hyper parameter tuning.

1 Like

Thanks Elahe for your help

1 Like