Hi,
I am creating a single layer network for graph embedding. I am trying to do a fast training with no_grad(). My embedding dimension is 35,128. My requirement is to select a random row from embedding.weight.data, calculation score, and multiply the scalar score to weight data of that particular vector or row.
You could sample a random number and index the embedding.weight directly via:
w = emb.weight[index]
Afterwards, you could calculate some score and reassign the new value to emb.weight[index].
I’m unsure, how the training works using random indices and skipping Autograd, so could you add a bit more information about your use case?
I am working on converting a c++ program which was used in a paper to convert nodes to vectors, so it can be used for ML task. We basically initialize the weight vector of size V x d (V - number of nodes, d - embedding size) with values between -0.5 to 0.5. Then we sample nodes as u, v and neg_v, where u & v has edge and u & neg_v does not have edge.
The calculation selects the weight of respective nodes and uses the below logic for calculating scores.
if we sample u, v:
score = -bias
score += weight[u] * weight[v]
score = (1 - fast_Sigmoid(score)) * lr
weight[u] += weight[v] * score
weight[v] += weight[u] * score
if we sample u, neg_v:
score = -bias
score += weight[u] * weight[neg_v]
score = - fast_Sigmoid(score) * lr
weight[u] += weight[neg_v] * score
weight[neg_v] += weight[u] * score
since all this are performed in single go and we want to have faster calculation, we came across with torch.no_grad() (" Disabling gradient calculation is useful for inference, when you are sure that you will not call backward(). it will reduce memory consumption") and we are using no_grad(). I just wanted to know if there is any way to select particular row of weight matrix and update it. I guess i got my answer from your reply. I am still trying to figure out how to multiple process this training along with memory sharing. If you can provide me some details it will be big help!!
Hope i gave a clear picture what i am trying to do!!
Is it possible to give your inputs for the above comment??