Embedding dimension error

1

I am trying to embed one-hot vectors, each with length 10000 and a corresponding label (0 or 1). With mini-batch size 10, the dimension of the input to my feedforward neural network model is 10 x 10000.

I am trying to embed this input with nn.Embedding(10000, 150).

So, after this embedding layer, my data becomes 10 x 10000 x 150. I am doing binary classification, so the output dimension is 10 x 10000 X 2. My label is of dimension 10 x 2. Either (1, 0) or (0, 1).

So, I get ValueError: Target size (torch.Size([10, 2])) must be the same as input size (torch.Size([10, 10000, 2]))

How can I make this model to work? Each vector of 10000 length corresponds to either a 0 or 1 for its label.

2

The error has to deal with the output layer not the embedding layer imo

3

Thanks for the reply. How would you suggest me to edit the final layer? What should the output dimension be?

Since the number of rows of my input data is 10000, I am not sure how the output layer does not include that number.

Here are the dimensions after each layer:


Dimension of input: torch.Size([10, 10000])
Dimension after embedding: torch.Size([10, 10000, 150])
Dimension after layer1: torch.Size([10, 10000, 512])
Dimension after layer2: torch.Size([10, 10000, 512])
Dimension after final layer: torch.Size([10, 10000, 2])
4

You need to add a Linear + Softmax layer on the output of the embedding layer.Here is a working example.

import os
import numpy as np
import torch
from torch import nn
from torch.autograd import Variable

class embedder(nn.Module):
  def __init__(self):
    super(embedder,self).__init__()
    self.embeddings=nn.Embedding(10000,150)
    self.l1=nn.Linear(150,1)
    self.sigm=nn.Sigmoid()

  def forward(self,x):
    x=self.embeddings(x)
    x=self.l1(x)
    x=self.sigm(x)
    return(x)

model=embedder()
criterion=nn.BCELoss()
optimizer=torch.optim.Adam(model.parameters(),lr=0.001)

inputIndices=np.random.randint(10000,size=(10000))
inputData=np.zeros((10000,10000))
for i,x in enumerate(inputIndices):
  inputData[i,x]=1
inputTarget=np.random.randint(2,size=(10000))

batchSize=10
epochs=10
numBatches=int(inputData.shape[0]/batchSize)

for curEpoch in range(epochs):
  totalLoss=0
  for curBatch in range(numBatches):
    model.zero_grad()
    X=Variable(torch.from_numpy(inputIndices[curBatch*batchSize:(curBatch+1)*batchSize].reshape(1,batchSize))).type(torch.long)
    Y=Variable(torch.from_numpy(inputTarget[curBatch*batchSize:(curBatch+1)*batchSize].astype(np.float32)))
    modelOutput=model(X)
    loss=criterion(modelOutput,Y)
    loss.backward()
    optimizer.step()
    totalLoss=totalLoss + loss.item()
  print("Epoch {0} Loss {1}".format(curEpoch,totalLoss))

Hope this helps

5

Yep as Anant said already, you have to do that;
Note Embedding layers just acts as a lookup table, they are moreover quite similar to one-hot-encoding as well