Hi, I’m trying to implement a partial AUC loss objective (see section 2.3 https://indico2.conference4me.psnc.pl/event/35/contributions/3501/attachments/1099/1141/Wed-3-2-1.pdf)
but I keep getting this error
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [128, 1]], which is output 0 of TBackward, is at version 2; expected version 1 instead.
I ran anomaly mode and it just points me to the output of the network, which I know isn’t the problem because it runs with BCE loss without any issues.
This is what I have so far for the implementation of the loss function:
class PartialAUCLoss(nn.Module):
def __init__(self, alpha=0, beta=0.1, gamma=5):
super(PartialAUCLoss, self).__init__()
self.alpha = alpha
self.beta = beta
self.gamma = gamma
self._delay = 100
self._idx_in_list = {}
self._neg_scores = None
def _get_range(self):
size = len(self._neg_scores)
n_alpha = int(-(-self.alpha*size // 1))
n_beta = int(self.beta*size // 1) - 1
return range(n_alpha, n_beta)
def step_func(self, val):
return 1/(1+torch.exp(-2*self.gamma*val))
def forward(self, x_scores, y, x_indices):
"""
:param x_scores: raw outputs from the NN for some batch (tensor)
:param y: labels for that batch (tensor)
:param x_indices: indices of for each sample in the batch (tensor)
"""
loss = 0
x_scores = x_scores.view(-1,)
y = y.view(-1,)
x_positives = x_scores[y==1]
x_negatives = x_scores[y==0]
x_neg_indices = x_indices[y==0].numpy()
if (len(x_positives) < 1) or (self._neg_scores is None) or (len(self._neg_scores) < self._delay):
loss = F.binary_cross_entropy(torch.sigmoid(x_scores), y)
else:
idx_range = self._get_range()
for x_pos in x_positives:
loss += self.step_func(x_pos-self._neg_scores[np.random.choice(idx_range, size=1)[0]])
if len(x_negatives) > 0:
include = []
for i, x_idx in enumerate(x_neg_indices):
if self._idx_in_list.get(x_idx, None) is None:
include.append(i)
self._idx_in_list[x_idx] = True # temorarily, to be used to handle updates later
else:
self._neg_scores[]
if include:
if self._neg_scores is not None:
self._neg_scores, _ = torch.sort(torch.cat(self._neg_scores, x_negatives[include]), descending=True)
else:
self._neg_scores, _ = torch.sort(x_negatives[include], descending=True)
return loss
Basically, I keep track of the negative samples, sort them, and compute the AUC loss from some random pair that’s within the desired range. Something here is causing the error to get thrown…
I also had to have loss.backward(retain_graph=True) in the batch loop.
Any idea on what’s going wrong here?
Note on implementation: For now I’m not updating the negative sample values since I still don’t know how to modify self._neg_scores in-place without getting an error.