Can't pickle local object 'DataLoader.__init__.<locals>.<lambda>'

vision
10 
        trsfm = transforms.Compose([
            transforms.Resize((32, 32)),
            transforms.ToTensor(),
            transforms.Normalize((0.1307,), (0.3081,)),
            transforms.Lambda(lambda x: x.repeat(3, 1, 1))
            ])

Should become

        def tmp_func(x):
            return x.repeat(3, 1, 1)

        trsfm = transforms.Compose([
            transforms.Resize((32, 32)),
            transforms.ToTensor(),
            transforms.Normalize((0.1307,), (0.3081,)),
            transforms.Lambda(tmp_func)
            ])
1 Like
11

Thank you so much man :slight_smile:
It’s working now.
Have a great day.

By the way can you tell me what’s the problem was before, why it can’t be pickled on windows but does on linux?

12

I have no idea :confused: That why I pinged peterjc which know windows much better.

13

Ok thank you for your help.
Let’s hope he replies

14

No, it is not supported on Windows. The reason is that multiprocessing lib doesn’t have it implemented on Windows. There are some alternatives like dill that can pickle more objects.

3 Likes
15

Could you please do me a favor and write this one into the official Windows doc?

16

Excuse my ignorance on the matter, but would you like to tell me how exactly can one use dill for .
I tried installing that too but that didn’t solve the problem either. May be I am not using it right and I have to make some changes in my code in order to use it.
Can you tell me about it’s usage to solve my problem?

17

You mean here:

Windows Discussion Forum
Or if I am wrong, would you share the link?

18

No, things should be done at the backend. I don’t think we will introduce dill to resolve this problem. So what @albanD said is the correct solution, at least for now. As for the document, it can be accessed here and modified here.

19

Hi,

Has there been any update on this in the official repo? I just ran into this issue on the latest Torch from Conda.

Cheers!

1 Like
20

Hi @albanD, I am facing the same issue while using Pool method. As you suggested, I removed lamda functions. Still the error is persisting. (I am using Windows, I used Colab to try the same code, but it is also showing the same error there … ) Can you suggest, what can be done?

21

Do you get a different error after removing the lambda? Because the error is explicitly about a lambda which cannot be serialized.

22

@albanD No, it’s the same error. I have also made sure that no lambda functions remain in the code.

23

If the error still points at a lambda function. Then you have one left. If it points to something else, you can share it here so that we can take a look.

1 Like
24

@albanD I have reproduced the error here : https://colab.research.google.com/drive/10Zxe40Tl14fWAaCAYg1fgQA13hHIiBo_?usp=sharing

The error is :

AttributeError                            Traceback (most recent call last)
<ipython-input-6-adca9abddcf0> in <module>()
    168 if __name__ == '__main__':
    169     t1 = time.time()
--> 170     main()
    171     print("Time taken :", time.time() - t1)

5 frames
<ipython-input-6-adca9abddcf0> in main(ways, shots, meta_lr, fast_lr, meta_batch_size, adaptation_steps, num_iterations, cuda, seed)
    126         args = [maml, tasksets]
    127         with Pool(4) as pool:
--> 128           values = pool.map(partial(ParallelTasks, args), list(range(meta_batch_size)))
    129 
    130         #for i in range(values):

/usr/lib/python3.6/multiprocessing/pool.py in map(self, func, iterable, chunksize)
    264         in a list that is returned.
    265         '''
--> 266         return self._map_async(func, iterable, mapstar, chunksize).get()
    267 
    268     def starmap(self, func, iterable, chunksize=None):

/usr/lib/python3.6/multiprocessing/pool.py in get(self, timeout)
    642             return self._value
    643         else:
--> 644             raise self._value
    645 
    646     def _set(self, i, obj):

/usr/lib/python3.6/multiprocessing/pool.py in _handle_tasks(taskqueue, put, outqueue, pool, cache)
    422                         break
    423                     try:
--> 424                         put(task)
    425                     except Exception as e:
    426                         job, idx = task[:2]

/usr/lib/python3.6/multiprocessing/connection.py in send(self, obj)
    204         self._check_closed()
    205         self._check_writable()
--> 206         self._send_bytes(_ForkingPickler.dumps(obj))
    207 
    208     def recv_bytes(self, maxlength=None):

/usr/lib/python3.6/multiprocessing/reduction.py in dumps(cls, obj, protocol)
     49     def dumps(cls, obj, protocol=None):
     50         buf = io.BytesIO()
---> 51         cls(buf, protocol).dump(obj)
     52         return buf.getbuffer()
     53 

AttributeError: Can't pickle local object 'omniglot_tasksets.<locals>.<lambda>'
25

@albanD The same error is seen in Linux environment too.

1 Like
26

GIven the names, I guess the problem is that the taskset you’re using cannot be serialized. And so you cannot use it in the process Pool.
You can either unpack these in a different object that you can serialize or change the libary to make the taskset serializable.

1 Like
27

Ah ok, I will try that approach. That’s for the insights @albanD. :grin:
That might be the case it seems.

28

I will post my solution here in case someone also faces the same error. As @albanD said, the error was because dataset couldn’t be serialized. I used import dill and the error was solved. In case someone is still facing issue, they can try this too from pathos.multiprocessing import ProcessingPool as Pool. :slight_smile:

29

Dear @Asura, in which file(s) specifically did you include the import dill statement?