Temperature Softmax implementation

nlp
1

I’m trying to implement a Softmax using temperature for an LSTM. This is what i came up with.

 out = model(out)
 _, idxs = out.max(1)
 # Apply temperature
soft_out = F.softmax(out / t, dim=1)
p = soft_out.data.cpu().numpy()
            
# Select a new predicted char with probability p
for j in range(soft_out.size()):
    idxs[j] = np.random.choice(out.size()[1], p=p[j])
    string += ix_to_char[idxs[j].data[0]]

Is this a correct approach?

2

This might help: https://github.com/szagoruyko/attention-transfer/blob/master/utils.py#L10