Vector view of model parameters?

danelliottster · June 6, 2017, 8:05pm

Is there a svelte way to obtain a vectorized view of the model parameters? Something that wouldn’t require a copy?

E.g. is there a way to concatenate the results of .view(-1) of all the parameters but for it to remain a view so no copy is done?

Thank you in advance.

smth · June 21, 2017, 11:28pm

you cannot do this for all model parameters, just because each parameter is on a separate storage. Concatenating them together will give you all of them together in a different storage (memory location).
This kind of flattening model parameters used to be a thing in (Lua)Torch, but we got rid of it.

What exactly is your use-case, maybe we can help find you a solution?

For something like LBFGS that needs a consistent single view, we have these helper functions, but they do things out of place (get a flattened view, then compute stuff, then push back gradients back):

github.com

pytorch/pytorch/blob/master/torch/optim/lbfgs.py#L57-L76


def _gather_flat_grad(self):
    views = []
    for p in self._params:
        if p.grad is None:
            view = p.data.new(p.data.numel()).zero_()
        elif p.grad.data.is_sparse:
            view = p.grad.data.to_dense().view(-1)
        else:
            view = p.grad.data.view(-1)
        views.append(view)
    return torch.cat(views, 0)


def _add_grad(self, step_size, update):
    offset = 0
    for p in self._params:
        numel = p.numel()
        # view as to avoid deprecated pointwise semantics
        p.data.add_(step_size, update[offset:offset + numel].view_as(p.data))
        offset += numel
    assert offset == self._numel()

danelliottster · June 30, 2017, 2:09pm

Thank you for the code snippet. That is the solution that I settled upon but, yes, I wanted something like what you described as being present only in Lua torch.

My use case is that I added an optimizer that wants all the parameters in vector form (ported from Matlab). Just wanted to avoid the copies.

What I have works for now. Maybe, between projects, I’ll hit you guys with a pull request with an attempt to re-add this functionality.