I would like to generate all combinations of elements in a list.I know I can use torch.combinations for that, but I need to use large parameters, which produces huge number of combinations. So the problem is that it does not fit into memory. For example :
The root of the problem is that something is wrong with how torch.combinations() is implemented. See this new thread
of mine and the github issue it references:
As an aside, generating all such combinations, either one at a time
or in “pages,” is straightforward enough just using python loops.
However, if torch.combinations()did work correctly and you
were working with a truly large problem, you could split your set
of elements into multiple subsets, apply torch.combinations()
on the subsets, and recombine the sub-combinations back into
combinations of elements of the original set. You could do this a
piece at a time (“pages”) so that you would never have to materialize
the entire (potentially very large) set of combinations all at once.