# Hyperparameter tuning using Bayesian optimization

**URL:** <https://discuss.pytorch.org/t/hyperparameter-tuning-using-bayesian-optimization/36145>\
**Category:** Uncategorized\
**Created:** [February 1, 2019, 12:40pm UTC](https://discuss.pytorch.org/t/hyperparameter-tuning-using-bayesian-optimization/36145 "2019-02-01T12:40:26Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Wafaa\_Wardah](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wafaa_wardah/32/53794_2.png) [@Wafaa\_Wardah](https://discuss.pytorch.org/u/Wafaa_Wardah)\
**Post date:** [February 1, 2019, 12:40pm UTC](https://discuss.pytorch.org/t/hyperparameter-tuning-using-bayesian-optimization/36145/1 "2019-02-01T12:40:26Z")

</div>

I am using CNN for a binary classification problem and need to use Bayesian optimization to tune parameters like learning rate, number of hidden layers, optimizers, etc

I have built my model with fixed learning rate and number of epochs and Adam optimizer. Can you please give me some guidance on where to go from here in order to have Bayesian optimization implemented for tuning these parameters? And how would I deal with my 3 data sets (training, validation and test)?

I will really appreciate your help.  
Thanks.

---

<div class="post-metadata">

**Author:** ![rasbt](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rasbt/32/60_2.png) [@rasbt](https://discuss.pytorch.org/u/rasbt)\
**Post date:** [February 1, 2019, 2:47pm UTC](https://discuss.pytorch.org/t/hyperparameter-tuning-using-bayesian-optimization/36145/2 "2019-02-01T14:47:11Z")

</div>

You don’t need to do anything special to perform bayesian optimization for your hyperparameter tuning when using pytorch. You could just setup a script with command line arguments like `--learning_rate`, `--num_layers` for the hyperparameters you want to tune and maybe have a second script that calls this script with the diff. hyperparameter values in your bayesian parameter optimization loop.

Conceptually, you can do sth like this

```auto
for round i in max_rounds:
     output = subprocess.call('model.py', '--learning_rate', 'some_val', '--num_layers', 'some_number')
     score = # parse score from output string (e.g., your loss output)
     #optimize p(score | hyperparam values)
     # get some_val, some_number for next round

```

But it is probably more efficient if you use a library like hyperopt ([https://github.com/hyperopt/hyperopt](https://github.com/hyperopt/hyperopt))

Maybe sth like

```auto

def run_model(learning_rate):
    # your model init here
    # your training here
    return loss

import numpy as np
from hyperopt import hp, tpe, fmin

# Single line bayesian optimization of polynomial function
best = fmin(fn=lambda x: run_model(x),
            space=hp.normal('x', 0.00001, 0.9),
            algo=tpe.suggest, 
            max_evals=2000)

```

But given how expensive typical model training for DL is, it may take a long time …

---

<div class="post-metadata">

**Author:** ![eph\_adm](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/90db22/32.png) [@eph\_adm](https://discuss.pytorch.org/u/eph_adm)\
**Post date:** [July 26, 2019, 1:54pm UTC](https://discuss.pytorch.org/t/hyperparameter-tuning-using-bayesian-optimization/36145/3 "2019-07-26T13:54:44Z")

</div>

I went through similar problem only mine was hyperparameter tuning for feedforward neural networks. Then I ran into this very helpful article, you should check it:

> **[Feedforward neural network & Optimization of Hyperparameters using...](https://blog.valcarce.fr/feedforward-neural-network/)**
>
> 0. Overview Here is shown how to create a Feedforward neural network (FNN) using Pytorch -- Facebook's AI framework. We'll go through the creation of a FNN, with a user set number of layer/neurones. Making a custom Dataloader is also discussed, so...

---

<div class="post-metadata">

**Author:** ![aksg87](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aksg87/32/15602_2.png) [@aksg87](https://discuss.pytorch.org/u/aksg87)\
**Post date:** [November 28, 2019, 12:40am UTC](https://discuss.pytorch.org/t/hyperparameter-tuning-using-bayesian-optimization/36145/4 "2019-11-28T00:40:19Z")

</div>

Looks like this link is down ☹

---

<div class="post-metadata">

**Author:** ![Naveen\_Kumar](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/naveen_kumar/32/5365_2.png) [@Naveen\_Kumar](https://discuss.pytorch.org/u/Naveen_Kumar)\
**Post date:** [November 28, 2019, 8:32am UTC](https://discuss.pytorch.org/t/hyperparameter-tuning-using-bayesian-optimization/36145/5 "2019-11-28T08:32:46Z")

</div>

Yup the link is down FYI @eph_adm

---

<div class="post-metadata">

**Author:** ![saba](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saba/32/19439_2.png) [@saba](https://discuss.pytorch.org/u/saba)\
**Post date:** [March 30, 2020, 3:38am UTC](https://discuss.pytorch.org/t/hyperparameter-tuning-using-bayesian-optimization/36145/6 "2020-03-30T03:38:42Z")

</div>

Hi All,

I want to do hyper parameter tuning for CNN layers ( 2 or 3 layers), number of filters for CNN, FC layers ( 2 or 3 layers) and number of neurons ([100:10:100]) , batch size {100,200}, LR {10^-4,10^-5}, Dropout{0.3,0.5,0.7}.

Do you know any Pytorch package to implement as grid search?

---

<div class="post-metadata">

**Author:** ![Elahe\_Heydari](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/elahe_heydari/32/13722_2.png) [@Elahe\_Heydari](https://discuss.pytorch.org/u/Elahe_Heydari)\
**Post date:** [July 11, 2020, 1:42pm UTC](https://discuss.pytorch.org/t/hyperparameter-tuning-using-bayesian-optimization/36145/7 "2020-07-11T13:42:42Z")

</div>

you can use [skorch](https://github.com/skorch-dev/skorch) for hyper parameter tuning.

---

<div class="post-metadata">

**Author:** ![saba](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saba/32/19439_2.png) [@saba](https://discuss.pytorch.org/u/saba)\
**Post date:** [July 11, 2020, 10:21pm UTC](https://discuss.pytorch.org/t/hyperparameter-tuning-using-bayesian-optimization/36145/8 "2020-07-11T22:21:22Z")

</div>

Thanks Elahe for your help  
🙂