Hyperparameter Search for MultiOutputRegression

1

Hello everybody,

just wondering if there are methods for hyperparameter search when using multioutput regression?
I tried skorch but its not possible to pass more than 1 output, so it makes not really sense for my project…

Thanks for the help

2

Not sure, if this is really a limitation of Skorch, but @ottonemo might know. :wink:

1 Like
3

Can you please elaborate what you mean with

its not possible to pass more than 1 output

It is certainly possible to do a multi-output regression with skorch alongside GridSearchCV. Minimal example:

X = ...
y = ...
assert y.shape[1] == 2

regressor = skorch.toy.make_regressor(output_units=y.shape[1])
net = skorch.NeuralNetRegressor(regressor)
gs = sklearn.model_selection.GridSearchCV(net, params={'module__dropout': [0, 0.1, 0.2]}, scoring='neg_mean_squared_error')
gs.fit(X, y)
1 Like
4

Hi Ottonemo,

My case is kind of similar like the one you discussed. But I want to wrap the LSTM model from PyTorch which is a multi-output regression. If that, how can I realize it?

Thanks

5

Yes, of course :slight_smile:

See this example:

# timeful multi regression
getX = lambda i: np.linspace(0+i, 10+i, 10)[None, :, None].astype(np.float32)
X = np.vstack([getX(i) for i in range(10)])
y = np.vstack([
    np.concatenate([Xi[None]**2, Xi[None]*3], axis=-1)
    for Xi in X
])

assert X.shape == (10, 10, 1)
assert y.shape == (10, 10, 2)

class RNNRegressor(torch.nn.Module):
    def __init__(self, dropout=0.):
        super().__init__()
        self.rnn = torch.nn.LSTM(input_size=1, hidden_size=10, batch_first=True)
        self.rnn.reset_parameters()
        self.dop = torch.nn.Dropout2d(p=dropout)
        self.out = torch.nn.Linear(10, 2)
        self.out.reset_parameters()
        
    def forward(self, X):
        out_rnn, _hid = self.rnn(X)
        out_lin = self.out(out_rnn)
        return out_lin
    
net = skorch.NeuralNetRegressor(RNNRegressor)
net.fit(X, y)

Of course, for grid search, you will need to define a custom split as with any other time-series data. But that’s not related to skorch and varies depending on what your data looks like.

6

Hi Ottonemo,

Thank you very much for the quick response. My training dataset is like this:

tensor([[ 50., 50., 50., 50.],
[ 50., 50., 50., 50.],
[ 50., 50., 50., 51.],
…,
[169., 169., 169., 170.],
[169., 169., 170., 170.],
[169., 170., 170., 170.]])
tensor([[ 50., 50., 50., 50.],
[ 50., 50., 50., 51.],
[ 50., 50., 51., 52.],
…,
[169., 169., 170., 170.],
[169., 170., 170., 170.],
[170., 170., 170., 170.]])

My main code is like this:

PyTorch LSTM regressor

class AirModel(nn.Module):
def init(self):
super().init()
self.lstm = nn.LSTM(input_size=1, hidden_size=50, num_layers=1, batch_first=True)
self.linear = nn.Linear(50, 1)
def forward(self, x):
x, _ = self.lstm(x)
x = self.linear(x)
return x

creat the skorch wrapper

model_skorch=NeuralNetRegressor(
module=AirModel,
criterion=nn.MSELoss,
optimizer=optim.Adam,
verbose=True)

param_grid={
‘batch_size’:[8,10],
‘max_epochs’:[500],
}

grid search in Scikit-learn

grid = GridSearchCV(estimator=model_skorch, param_grid=param_grid, n_jobs=-1, cv=3,error_score=‘raise’)
grid_result = grid.fit(X_train, y_train)

But there is always a error in the last row of code: “input.size(-1) must be equal to input_size. Expected 1, got 4”

I appreciate any comments about it