# I am lost on custom batch size definition

**URL:** <https://discuss.pytorch.org/t/i-am-lost-on-custom-batch-size-definition/179797>\
**Category:** data\
**Created:** [May 12, 2023, 7:07pm UTC](https://discuss.pytorch.org/t/i-am-lost-on-custom-batch-size-definition/179797 "2023-05-12T19:07:30Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![gBrecht](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gbrecht/32/60449_2.png) [@gBrecht](https://discuss.pytorch.org/u/gBrecht)\
**Post date:** [May 12, 2023, 7:07pm UTC](https://discuss.pytorch.org/t/i-am-lost-on-custom-batch-size-definition/179797/1 "2023-05-12T19:07:30Z")

</div>

Hallo,

I have a problem understanding how I can define a Dataset/DataLoader combination that processes batches of a custom size. I have a tabular dataset with a categorical variable defining the batch. I define the dataset like

```auto
class MyDataset:

    def __init__ (self, df, features, target):
        self.df = df
        self.features = features
        self.target = target
        self.category_var = list(self.df.category_var.unique())

    def __len__ (self):
        return len(self.category_var)

    def __getitem__ (self, idx):
        var = self.category_var[idx]
        X = self.df.query('category_var==@var')[self.features]
        y = self.df.query('category_var==@var')[self.target]
        return X,y

```

So each item in my dataset is a custom sized batch of samples I want to process.

When I defined a train\_loader like

```auto
train_loader = torch.utils.data.DataLoader(
    MyDataset(df, features, target), batch_size=None, batch_sampler=None
)

```

My code tanks with:

`TypeError: 'int' object is not callable`

Which does not give me an angle to work with.

I guess I am understanding the concept of custom dataset and dataloaders, maybe even the definition of a batch wrong.  
Is my dataset meant to return a batch or a sample?  
If sample: How can my dataset define a custom batch size if it only returns one sample? Is that not the job of the dataset? if not, whose is it? the data loaders? the collate\_fn?

Thank you very much

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [May 13, 2023, 2:11pm UTC](https://discuss.pytorch.org/t/i-am-lost-on-custom-batch-size-definition/179797/2 "2023-05-13T14:11:18Z")

</div>

> [@gBrecht](#):
>
> Is my dataset meant to return a batch or a sample?

In the standard use case your ` __getitem__ ` method will load, process, and return a single sample.

> [@gBrecht](#):
>
> If sample: How can my dataset define a custom batch size if it only returns one sample?

I’m unsure what “custom” batch size means. If you want to set the batch size to a specific value you could do so in the `DataLoader` by setting the `batch_size` argument.

---

<div class="post-metadata">

**Author:** ![gBrecht](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gbrecht/32/60449_2.png) [@gBrecht](https://discuss.pytorch.org/u/gBrecht)\
**Post date:** [May 14, 2023, 6:36am UTC](https://discuss.pytorch.org/t/i-am-lost-on-custom-batch-size-definition/179797/3 "2023-05-14T06:36:12Z")

</div>

Thank you very much for your reply.  
I do not mean setting a custom `batch_size` argument, because that would be fixed for all batches. I want every one of my batches to have a different size.

In my example I have a variable (called `category_var`) which groups the data into batches and my ` __getitem__ ` wrongly returns one of these batches.

As `Dataset. __getitem__ ` is not the place to do this. Do you have a hint on how to achieve this?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [May 14, 2023, 6:28pm UTC](https://discuss.pytorch.org/t/i-am-lost-on-custom-batch-size-definition/179797/4 "2023-05-14T18:28:24Z")

</div>

I think `Dataset. __getitem__ ` could still be used to load multiple samples and you could then skip the batching from the `DataLoader` by setting `batch_size=1` during its initialization.  
I’m currently unsure what exactly is failing as your initial post doesn’t mention the line of code using your approach.

---

<div class="post-metadata">

**Author:** ![gBrecht](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gbrecht/32/60449_2.png) [@gBrecht](https://discuss.pytorch.org/u/gBrecht)\
**Post date:** [May 14, 2023, 6:56pm UTC](https://discuss.pytorch.org/t/i-am-lost-on-custom-batch-size-definition/179797/5 "2023-05-14T18:56:15Z")

</div>

thx again. Do you know of a minimal working example (kind of the opposite of what you ask from me…) I could start off of? I really struggle to wrap my head around the interaction of Dataset, Dataloader, Sampler and Batches.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [May 14, 2023, 7:15pm UTC](https://discuss.pytorch.org/t/i-am-lost-on-custom-batch-size-definition/179797/6 "2023-05-14T19:15:14Z")

</div>

Sure, here is a simple example which uses pre-defined batch sizes in the `Dataset`:

```python
class MyDataset(Dataset):
    def __init__ (self):
        self.data = torch.randn(100, 1)
        self.target = torch.randint(0, 10, (100,))
        self.batch_sizes = [10, 20, 50, 15, 5]
        assert sum(self.batch_sizes) == self.data.size(0)
        
    def __len__ (self):
        return len(self.batch_sizes)
    
    def __getitem__ (self, index):
        batch_size = self.batch_sizes[index]
        offset = sum(self.batch_sizes[:index])
        print(f"loading {batch_size} samples at offset {offset}")
        x = []
        y = []
        for i in range(batch_size):
            x.append(self.data[offset+i])
            y.append(self.target[offset+i])
        x = torch.stack(x)
        y = torch.stack(y)
        return x, y

dataset = MyDataset()
loader = DataLoader(dataset, batch_size=1, shuffle=True)

for data, target in loader:
    # remove batch dimension added by the DataLoader
    data.squeeze_(0)
    target.squeeze_(0)
    print(data.shape, target.shape)
    
# loading 15 samples at offset 80
# torch.Size([15, 1]) torch.Size([15])
# loading 10 samples at offset 0
# torch.Size([10, 1]) torch.Size([10])
# loading 50 samples at offset 30
# torch.Size([50, 1]) torch.Size([50])
# loading 20 samples at offset 10
# torch.Size([20, 1]) torch.Size([20])
# loading 5 samples at offset 95
# torch.Size([5, 1]) torch.Size([5])

```

You could also take a look at the `BatchSampler` approach if it would fit your use case better as described in [this post](https://discuss.pytorch.org/t/how-to-use-batchsampler-with-getitem-dataset/78788/4).

---

<div class="post-metadata">

**Author:** ![gBrecht](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gbrecht/32/60449_2.png) [@gBrecht](https://discuss.pytorch.org/u/gBrecht)\
**Post date:** [May 14, 2023, 7:48pm UTC](https://discuss.pytorch.org/t/i-am-lost-on-custom-batch-size-definition/179797/7 "2023-05-14T19:48:53Z")

</div>

Thank you very much!! I will try it out and hopefully report success. Just looking at it I fail to see how this dataset will work as it does not return a single item in **getitem** and therefore should fail the same as mine …

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [May 15, 2023, 4:47am UTC](https://discuss.pytorch.org/t/i-am-lost-on-custom-batch-size-definition/179797/8 "2023-05-15T04:47:10Z")

</div>

You still didn’t explain where exactly the error message is raised and since my code snippet is executable and not failing I also don’t know why you think it should fail.

---

<div class="post-metadata">

**Author:** ![gBrecht](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gbrecht/32/60449_2.png) [@gBrecht](https://discuss.pytorch.org/u/gBrecht)\
**Post date:** [May 15, 2023, 6:39am UTC](https://discuss.pytorch.org/t/i-am-lost-on-custom-batch-size-definition/179797/9 "2023-05-15T06:39:13Z")

</div>

You are right. This is the “minimal” example I would love to get working and understand why it does:  
The current error is a little bit different than the one I posted as I tinkered a little bit.

```auto
import torch
import numpy as np
import pandas as pd
import random

data = np.random.rand(100, 10)
df = pd.DataFrame(data, columns=[f'{i}' for i in range(10)])
df["cat_var"] = [random.choice([f"batch_{i+1}" for i in range(5)]) for j in range(100)]
device = "cpu"

class MyDataset:
    def __init__ (self, df, features, target, cat_var):
        self.df = df
        self.features = features
        self.target = target
        self.category_var = list(df.cat_var.unique())

    def __len__ (self):
        return len(self.category_var)

    def __getitem__ (self, idx):
        print(idx)
        var = self.category_var[idx]
        X = torch.tensor(self.df.query("cat_var==@var")[self.features].values)
        y = torch.tensor(self.df.query("cat_var==@var")[self.target])
        return X, y

dataset = MyDataset(df, [f'{i}' for i in range(9)], '9', "cat_var")
train_loader = torch.utils.data.DataLoader(dataset)

class Model(torch.nn.Module):
    def __init__ (self):
        super(Model, self). __init__ ()
        self.layer = torch.nn.Linear(in_features=9, out_features=1)

    def forward(self, x):
        return self.layer(x)

model = Model().to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
loss = torch.nn.MSELoss()
epoch_losses = []
for epoch in range(5):
    epoch_loss = 0
    for X, y in train_loader:
        X, y = X.to(device), y.to(device)
        pred = model(X)
        l = loss(pred, y)
        epoch_loss += l.item()
        optimizer.zero_grad()
        l.backward()
        optimizer.step()

```