Lookup table for loss function?

bogdanstate · June 8, 2017, 9:47am

I am trying to use a beta-binomial loss, and I am running into all sorts of pain related to the computation of the gradient on the gamma function (a separate discussion has happened elsewhere on this forum).

Thinking about it a little, it occurs to me that since the Beta function is quite smooth, it should be possible to compute a gradient for a pre-specified set of data points, and use that to interpolate the gradient values at runtime. This is especially true since for large values of the parameters (a, b) the Beta distribution approximates well to a Gaussian.

So, my question then is, is there anything wrong with my understanding here? I am asking since I am relatively new to PyTorch and deep learning. I assume a lookup table strategy would be easy to implement using tensor operations in Python directly. And I am also wondering if nn.LookupTable couldn’t be repurposed to this end. Just wanted to check before going down this path, thanks!

smth · June 22, 2017, 3:27am

yes you can do exactly this. You can use nn.LookupTable for this purpose, but it’s easier to simply write your own cute autograd.Function, like this:

github.com

pytorch/pytorch/blob/master/torch/autograd/_functions/basic_ops.py#L41-L54


import torch
from ..function import Function, traceable
import math




def sort_args(a, b, key=torch.is_tensor):
return (a, b, True) if key(a) else (b, a, False)




@traceable
class PowConstant(Function):


@staticmethod
def forward(ctx, a, b):
    tensor, ctx.constant, ctx.tensor_first = sort_args(a, b)
    if ctx.tensor_first:
        ctx.save_for_backward(tensor)
        return tensor.pow(ctx.constant)
    else:
        result = torch.pow(ctx.constant, tensor)

This file has been truncated. show original