yes on windows it can’t be be at the moment.
Do you know any solutions?
So they can be pickled?
The error then does not come from the fact that you try to pickle a lambda function?
Sorry I miss-typed, my earlier comment is edited
I guess you will need to change NotMNISTDataLoader
and use a proper function there instead of a lambda.
so, for my problem:
transforming uni-channeled image into multi-channeled tensor, what can I do?
because this works fine on linux, but for windows users what’s the alternative.
for example if my batch size is tensor(128, 32, 32), after transformation I want tensor(128, 3, 32, 32)
Do you have the code for the __init__
of NotMNISTDataLoader
? There is a lamda function in there like lambda x: x+1
that needs to be changed to
def tmp_func(x):
return x + 1
And replace the lambda by tmp_func
.
Here you go:
class NotMNISTDataLoader(BaseDataLoader):
"""
NotMnist data loading demo using BaseDataLoader
"""
def __init__(self, data_dir, batch_size, shuffle, validation_split, num_workers, training=True):
print("NotMnist is used")
self.data_dir = data_dir + "NotMNIST/"
trsfm = transforms.Compose([
transforms.Resize((32, 32)),
transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,)),
transforms.Lambda(lambda x: x.repeat(3, 1, 1))
])
self.dataset = NotMNIST(self.data_dir, train = training, download = True, transform = trsfm)
super(NotMNISTDataLoader, self).__init__(self.dataset, batch_size, shuffle, validation_split, num_workers)
trsfm = transforms.Compose([
transforms.Resize((32, 32)),
transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,)),
transforms.Lambda(lambda x: x.repeat(3, 1, 1))
])
Should become
def tmp_func(x):
return x.repeat(3, 1, 1)
trsfm = transforms.Compose([
transforms.Resize((32, 32)),
transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,)),
transforms.Lambda(tmp_func)
])
Thank you so much man
It’s working now.
Have a great day.
By the way can you tell me what’s the problem was before, why it can’t be pickled on windows but does on linux?
I have no idea That why I pinged peterjc which know windows much better.
Ok thank you for your help.
Let’s hope he replies
No, it is not supported on Windows. The reason is that multiprocessing lib doesn’t have it implemented on Windows. There are some alternatives like dill that can pickle more objects.
Could you please do me a favor and write this one into the official Windows doc?
Excuse my ignorance on the matter, but would you like to tell me how exactly can one use dill for .
I tried installing that too but that didn’t solve the problem either. May be I am not using it right and I have to make some changes in my code in order to use it.
Can you tell me about it’s usage to solve my problem?
No, things should be done at the backend. I don’t think we will introduce dill to resolve this problem. So what @albanD said is the correct solution, at least for now. As for the document, it can be accessed here and modified here.
Hi,
Has there been any update on this in the official repo? I just ran into this issue on the latest Torch from Conda.
Cheers!
Hi @albanD, I am facing the same issue while using Pool method. As you suggested, I removed lamda functions. Still the error is persisting. (I am using Windows, I used Colab to try the same code, but it is also showing the same error there … ) Can you suggest, what can be done?
Do you get a different error after removing the lambda? Because the error is explicitly about a lambda which cannot be serialized.