Use torch.combinations with large parameters

[KFrank]

Hi Dofasol!

@dofasol

So the problem is that it does not fit into memory.

The root of the problem is that something is wrong with how
torch.combinations() is implemented. See this new thread
of mine and the github issue it references:

Egregious inefficiency in torch.combinations()

Hi All! This post adds context and details to the existing pytorch github issue 41325 and a recent forum thread (see below). torch.combinations() breaks on problems of moderate size and is vastly slower than other implementations, such as python’s itertools.combinations(). One oddity (as noted in the github issue) is that torch.combinations (elements, 1) can work, where the essentially equivalent computation, torch.combinations (elements, len (elements) - 1), fails. Consider this scri…

Is there a way to make kind of a pagination with this function ?

As an aside, generating all such combinations, either one at a time
or in “pages,” is straightforward enough just using python loops.

However, if torch.combinations() did work correctly and you
were working with a truly large problem, you could split your set
of elements into multiple subsets, apply torch.combinations()
on the subsets, and recombine the sub-combinations back into
combinations of elements of the original set. You could do this a
piece at a time (“pages”) so that you would never have to materialize
the entire (potentially very large) set of combinations all at once.

Best.

K. Frank