How can I apply k-fold cross validation with CNN. I do not want to make it manually; for example, in leave one out, I might remove one item from the training set and train the network then apply testing with the removed item. Could you please help me to make this in a standard way.
It seems a call to .fit re-initializes the model when warm_start is set to false. The model reset should therefore be performed automatically in grid searches.
@ptrblck, thanks for the clue of using skorch. I am new to skorch and pytorch. My understanding is that skorch is a pytorch wrapper for sklearn. I found sklearn seems not to support CNN by searching online. For example, cross_val_score need sklearn estimators. sklearn model_selection fit function need X: array-like, shape = [n_samples, n_features] instead of images. Please help me understand how skorch can help cross validation for CNN. Thanks.
Skorch tries to add exactly the missing compatibility of scikit-learn for PyTorch.
The docs have some good examples.
E.g. using the NeuralNet class you can call methods like fit and predict using PyTorch models in the background.
@ptrblck Is there specific way to put our X, y data (generated as training and validation Loaders already) in the grid search fit method directly? For instance what is the preferred X, y input for the fit function in this scenario? It fails when I Provide the generated training data Loader (images directly from directory) in the grid search fit function. some linky examples if there are, will be quite helpful. Thanks
A question for cross-validation.
Firstly, we divide all the data into training samples and test samples, such as the proportion of 80% and 20%.
Then, we divide the training samples into five groups, four of which used as train data (64%) and one group used as validate data (16%). The 5-fold cross-validation can be carried out to find the suitable parameters of the CNN.
So after the 5-fold cross validation, what should we do next for the testing samples? Should we use the parameters of CNN that determined in the cross validation stage, to train the network again on the 80% samples, and test on the 20% samples?
I am a little confused. Thank you.
Thanks for the reply @ptrblck. It actually works with my dataset (dataSet from ImageFolder with transformations applied), but in the forward(X) method of the neural net, It passes a List of X, y tensors, instead of just X tensors, which is weird. I do Grid Search with Skorch, and here is the sample code:
The skorch.dataset.Dataset implementation is not the same as torch.utils.data.Dataset as described here. Could you try to wrap your data in the skorch implementation and pass it to the fit function?
IIt is unclear for me yet of how to use Pytorchs DataLoader and Sklearns Cross validation? All I can find in Inet is: it is impossible, Skorch`s explanation is not perspicuously for me.
Could you give an advise of wrapping DataLoader in Skorch in this way?
Thank you a lot in advance!