How could i change my output size as i expected?

1

hello all i am a beginner in deep learning and pytorch. I have made sequential model in pytorch like code below.
input shape : (1934,1024)
expected output shape : (1934,8)
batch size = 32

when i train my model and check the output the size turn out to be (14,8). How i can have my output size as my expected size

class MLP(nn.Module):
    def __init__(self):
        super(MLP, self).__init__()
        self.layers = nn.Sequential(
            nn.Linear(1024, 100),
            nn.Sigmoid(),
            nn.Linear(100, 8),
            #tempsigmoid()
            TempSigmoid()
            )
        
    def forward(self, x):
        x = self.layers(x)
        return x

this is how i train my model

# Model TRAINING
epochs = 5
train_losses=[]
elapsed=[]

for epochs in range(epochs):
    t0 = time.time()
    model.train()
    for batch_idx, (x,y) in enumerate(loader):
        x,y = x.to(device), y.to(device)
        optimizer.zero_grad()
        outputs = model(x)
        loss = loss_fn(outputs,y)
        loss.backward()
        optimizer.step()
        StopatLossValue()
        
        # program timing
        torch.cuda.synchronize() #the right way to measure time in pytorch
        t1 = time.time()
        eta = t1-t0
        
        if batch_idx % 100 == 0:
            print('Train Epoch: {} \t Loss: {:.6f} \t Elapsed time :{:.2f} ms'.format(epochs, loss.item(), (eta*1000)))
            elapsed.append(float(eta))
            np.savetxt("eta.csv", elapsed, delimiter=",")
2

Well, As far as I can see you should get the proper size. What is TempSigmoid? is it a pytorch’s function. That’s probably what is reducing your output size.

3

i was custom sigmoid function with tempsigmoid

class TempSigmoid(nn.Module):
    def __init__(self):
        super(TempSigmoid, self).__init__() # writing super in python 2.7 (difference with python 3)
        #self.nd = nd
        global nd # define global variable in python
        nd = 3.0
  
    def forward(self,x):
        temp=nd/torch.log(torch.tensor(0.9))
        return torch.sigmoid(x/(temp))

But i think it doesn’t effect the outputs size, because when i checked by not using tempsigmoid its still the same size (14,8)

4

I’m afraid your input is not the size you think :frowning:

import torch
from torch import nn



class MLP(nn.Module):
    def __init__(self):
        super(MLP, self).__init__()
        self.layers = nn.Sequential(
            nn.Linear(1024, 100),
            nn.Sigmoid(),
            nn.Linear(100, 8),
            # tempsigmoid()
        )

    def forward(self, x):
        x = self.layers(x)
        return x

a= torch.rand(32,1934,1024)

model = MLP()
out = model(a)
print(out.shape)

torch.Size([32,1934, 8])

Btw I didn’t use TempSigmoid as temp variable is not defined

5

so the problem is my input data is not as i am thinking?
Below is my dataset .

x = np.loadtxt("./input/set3/new_set3.csv", delimiter=",")
y = np.loadtxt("./input/set3/ideal_full.csv", delimiter=",")

x_train = torch.from_numpy(x).float()
y_train = torch.from_numpy(y).float()


dataset = torch.utils.data.TensorDataset(x_train, y_train)
loader = DataLoader(
    dataset=dataset,
    batch_size = 32,
    shuffle=True,
    num_workers=0,
    pin_memory=torch.cuda.is_available()
)

I check when prepare dataset my input still (1934,1024). I’ve checked too in training loop after model.train() still the same size, after that it change to (14,1024) what causing it changed?

6

I have no idea as I have no acess to your dataset
I would iterate over the dataset asserting the size

dataset = torch.utils.data.TensorDataset(x_train, y_train)
loader = DataLoader(
    dataset=dataset,
    batch_size = 32,
    shuffle=True,
    num_workers=0,
    pin_memory=torch.cuda.is_available()
)
for sample in loader:
    assert sample.shape == expected

But for sure you have some sneaky line somewhere you forgot.

8

i had solved it by changing my batch size to 1934 the same number with my samples

9

I don’t know It shouldn’t have any effect.
Linear layer takes a (*,N) input. At least in the later versions.

10

i dont know why it works. i am using torch 1.2.0

11

that would mean that you only have a single batch now, isn’t effective as this BS might hinder your performance as well…

12

excuse for my not understanding. So this batch size make my model performance not good?

13

it depends on your task at hand as well; I am . pretty sure you have some data from csv’s probably; Had it been some image data etc, the model might take a longer time to generalize [speaking in general], plus everytime we don’t have the possibility to increase the batch size to 1k+ as well; (system constraints)
I would suggest, try to narrow down things with the example you have first :slight_smile:

14

i am a beginner in DL and pytorch and doing research with custom activation function and weight clip and stop training with loss value.

My input is a handwriting number with 32x32 and has 1934 samples number. and my target data is 1934 samples with 8 bit so i need output with 1934 and 8 bit so i can compare and count loss with my target data.

this is the first time to i am using batch size to 1k+ and i dont know whether is it right or no.

do you have suggestion the good example which i can learn. I am googling all and make my model just like now

15

Here you have a fully runnable script which fits exactly what you specified.

import torch
from torch import nn


class TempSigmoid(nn.Module):
    def __init__(self):
        super(TempSigmoid, self).__init__()  # writing super in python 2.7 (difference with python 3)
        self.nd =3.0


    def forward(self, x):
        temp = self.nd / torch.log(torch.tensor(0.9))
        return torch.sigmoid(x / (temp))

class MLP(nn.Module):
    def __init__(self):
        super(MLP, self).__init__()
        self.layers = nn.Sequential(
            nn.Linear(1024, 100),
            nn.Sigmoid(),
            nn.Linear(100, 8),
            # tempsigmoid()
            TempSigmoid()
        )

    def forward(self, x):
        x = self.layers(x)
        return x

N = 10
epochs = 1

inp= torch.rand(N*32,1934,1024)
gt= torch.rand(N*32,1934,8)
dataset = torch.utils.data.TensorDataset(inp,gt)
loader = torch.utils.data.DataLoader(
    dataset=dataset,
    batch_size = 32,
    shuffle=True,
    num_workers=0,
)
model = MLP()
loss_fn = torch.nn.L1Loss()
optimizer = torch.optim.SGD(model.parameters(),lr=0.1)
for epoch in range(epochs):
    model.train()
    for batch_idx, (x, y) in enumerate(loader):
        optimizer.zero_grad()
        outputs = model(x)
        loss = loss_fn(outputs, y)
        loss.backward()
        optimizer.step()
        print('Input shape: %r. Output shape %r' %(x.shape,outputs.shape))

Input shape: torch.Size([32, 1934, 1024]). Output shape torch.Size([32, 1934, 8])
Input shape: torch.Size([32, 1934, 1024]). Output shape torch.Size([32, 1934, 8])
Input shape: torch.Size([32, 1934, 1024]). Output shape torch.Size([32, 1934, 8])
Input shape: torch.Size([32, 1934, 1024]). Output shape torch.Size([32, 1934, 8])
Input shape: torch.Size([32, 1934, 1024]). Output shape torch.Size([32, 1934, 8])
Input shape: torch.Size([32, 1934, 1024]). Output shape torch.Size([32, 1934, 8])
Input shape: torch.Size([32, 1934, 1024]). Output shape torch.Size([32, 1934, 8])
Input shape: torch.Size([32, 1934, 1024]). Output shape torch.Size([32, 1934, 8])
Input shape: torch.Size([32, 1934, 1024]). Output shape torch.Size([32, 1934, 8])
Input shape: torch.Size([32, 1934, 1024]). Output shape torch.Size([32, 1934, 8])

I have to insist that if it’s not working you probably has any issue in your data. Mainly because linear layer takes a …,Q tensor and map it to a …,Q’.
Thus, the mistake is in the data or any line of code
Ofc, the suitability of the chosen architecture depends on what kinda data are you using.
In general the higher batch size, the better.

The thing is that, according to what you say here

My input is a handwriting number with 32x32 and has 1934 samples number. and my target data is 1934 samples with 8 bit so i need output with 1934 and 8 bit so i can compare and count loss with my target data.

I think Your input is not 1934,1024 but you have 1934 samples. That’s what’s not clear to me. Maybe it’s related to that.
Realize x and y (in your code) must be shape shape everywhere but the last dimension (depending on the loss you are using).

Can you print x.shape, y.shape, x_train.shape and y_train.shape

16

Thank you for your reply

I tried the code that you write and the result like below

Input shape: torch.Size([32, 1024]). Output shape torch.Size([32, 8])
Input shape: torch.Size([32, 1024]). Output shape torch.Size([32, 8])
Input shape: torch.Size([32, 1024]). Output shape torch.Size([32, 8])
Input shape: torch.Size([32, 1024]). Output shape torch.Size([32, 8])
Input shape: torch.Size([32, 1024]). Output shape torch.Size([32, 8])
Input shape: torch.Size([32, 1024]). Output shape torch.Size([32, 8])
Input shape: torch.Size([32, 1024]). Output shape torch.Size([32, 8])
Input shape: torch.Size([32, 1024]). Output shape torch.Size([32, 8])
Input shape: torch.Size([32, 1024]). Output shape torch.Size([32, 8])
Input shape: torch.Size([32, 1024]). Output shape torch.Size([32, 8])
Input shape: torch.Size([32, 1024]). Output shape torch.Size([32, 8])
Input shape: torch.Size([32, 1024]). Output shape torch.Size([32, 8])
Input shape: torch.Size([32, 1024]). Output shape torch.Size([32, 8])
Input shape: torch.Size([32, 1024]). Output shape torch.Size([32, 8])
Input shape: torch.Size([32, 1024]). Output shape torch.Size([32, 8])
Input shape: torch.Size([32, 1024]). Output shape torch.Size([32, 8])
Input shape: torch.Size([14, 1024]). Output shape torch.Size([14, 8])

in preparing my dataset i had print the shape

((1934, 1024), (1934, 8), (1934, 1024), (1934, 1024))

data shape are changing when it come to loop in training

17

I don’t know what those shapes corresponds to but it seems you did not properly set something right? dataset maybe

18

thank you for your reply, yes i think something its not right and still try to analyze it

19

I noticed that 1934%32=14, I think it is because you just output the last batch.

20

thank you for your respond. excuse me for my lack of knowledge , if i want to get all the output how i can do it?