I am using CNN for a binary classification problem and need to use Bayesian optimization to tune parameters like learning rate, number of hidden layers, optimizers, etc
I have built my model with fixed learning rate and number of epochs and Adam optimizer. Can you please give me some guidance on where to go from here in order to have Bayesian optimization implemented for tuning these parameters? And how would I deal with my 3 data sets (training, validation and test)?
You don’t need to do anything special to perform bayesian optimization for your hyperparameter tuning when using pytorch. You could just setup a script with command line arguments like --learning_rate, --num_layers for the hyperparameters you want to tune and maybe have a second script that calls this script with the diff. hyperparameter values in your bayesian parameter optimization loop.
Conceptually, you can do sth like this
for round i in max_rounds:
output = subprocess.call('model.py', '--learning_rate', 'some_val', '--num_layers', 'some_number')
score = # parse score from output string (e.g., your loss output)
#optimize p(score | hyperparam values)
# get some_val, some_number for next round
def run_model(learning_rate):
# your model init here
# your training here
return loss
import numpy as np
from hyperopt import hp, tpe, fmin
# Single line bayesian optimization of polynomial function
best = fmin(fn=lambda x: run_model(x),
space=hp.normal('x', 0.00001, 0.9),
algo=tpe.suggest,
max_evals=2000)
But given how expensive typical model training for DL is, it may take a long time …
I went through similar problem only mine was hyperparameter tuning for feedforward neural networks. Then I ran into this very helpful article, you should check it:
I want to do hyper parameter tuning for CNN layers ( 2 or 3 layers), number of filters for CNN, FC layers ( 2 or 3 layers) and number of neurons ([100:10:100]) , batch size {100,200}, LR {10^-4,10^-5}, Dropout{0.3,0.5,0.7}.
Do you know any Pytorch package to implement as grid search?
