- There are many ways of doing this
import numpy as np
import torch.nn as nn
def __init__(self, values):
self.values = values
for i in self.values:
a = ValueChoice([1, 2, 3])
i = iter(a)
print(nn.Conv2d(next(i), 2, 3)) # Conv2d(2, 2, kernel_size=(3, 3), stride=(1, 1))
print(nn.Conv2d(next(i), 2, 3)) # Conv2d(3, 2, kernel_size=(3, 3), stride=(1, 1))
print(nn.Conv2d(next(i), 2, 3)) # Conv2d(1, 2, kernel_size=(3, 3), stride=(1, 1))
print(nn.Conv2d(next(i), 2, 3)) # this will give StopIteration error: not enough values.
However, this will do what you ask, but not what you want. If you have something like
kernel = ValueChoice([3, 5, 7])
out_ch = ValueChoice([16, 32, 64])
You may want to search all possible permutations of these values, and my solution won’t do that.
Instead of coding something yourself I would suggest you use a library for hyperparameter tuning, something like that Tune: Scalable Hyperparameter Tuning — Ray v2.0.0.dev0
Reinventing the wheel can be very fun and instructive, but also error prone and inefficient in the long run
- You can find some ways of doing that, but not easily and probably it wouldn’t make much sense. The kernels have different size, so it isn’t clear how one would translate in the other.
If however you have a different number of filters, let’s say
nn.Conv2d(3, 16, 3) and
nn.Conv2d(3, 5, 3) and want to copy the first into the second, you could copy 5 random sampled filter from the
nn.Conv2d(3, 16, 3) into
nn.Conv2d(3, 5, 3). However, again, it seems quite a weird policy and I don’t personally think it’s a fruitful decision. But you are free to try.
Also, doing this copy is not going to be easy. Standard
state_dict to check that all values matches, and in your case it will throw an error because, well, they don’t! So you need to write your own copy function. You will need to access the parameters through
module.named_parameters() and then manually copying them.