Using IterableDataset, resulting AttributeError: 'list' object has no attribute 'dim'

data
1

I am new to pytorch and have some issue to with my first build.
I am trying to build a simple ANN by feeding in data through IterableDataset.
However, I get an error with [AttributeError: ‘list’ object has no attribute “dim”]
I guess I must have done something wrong with the output of IterableDataset.
How can I correct the code?
I have to use some method to not load my csv in one go as it is too large for the memory.

Here is my code:

class CustomIterableDatasetv1(IterableDataset):
    
    def __init__(self, filename):
        self.filename = filename

    def line_mapper(self, line):
        #Splits the line into text and label
        tmp = line.strip('\n').split(',')
        feature = tmp[:353]
        label  = tmp[-1]
        return feature, label


    def __iter__(self):
        file_itr = open(self.filename)
        mapped_itr = map(self.line_mapper,file_itr)
        
        return mapped_itr

params = {'batch_size': 64,
          'num_workers': 6}
max_epochs = 20

dataset = CustomIterableDatasetv1('pipeline2.csv')
dataloader = DataLoader(dataset, **params)

model = ANNnetwork()
criterion = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

for epoch in range(max_epochs): 
      for (features, label) in dataloader:
        
        optimizer.zero_grad()

        outputs = model(features)
        loss = criterion(outputs, label.squeeze(1))
        
        loss.backward()
        optimizer.step()
        cost = loss.item()
        if i % 100 == 0:
            print('Epoch:' + str(epoch) + ", Iteration: " + str(i) 
                  + ", training cost = " + str(cost))
2

It looks like you are providing a Python list when it is expecting a Tensor, can you check the line where the error is raised to see what the inputs’ types are?

You should also check what your __iter__ function is returning. Automatic batching will take 64 (your batch_size) of those and pass them into the default_collate function. Make sure that output has the type that your model expects. You can override collate_fn as well if the default function is not doing what you’d like. You can read more about automatic batching here.

3

Python lists cannot be divided into separate lists based on characters that appear in the values of a list. This is unlike strings which values can be separated into a list. The AttributeError is an exception thrown when an object does not have the attribute you tried to access. The ‘list’ object has no attribute ‘split’ and you’re trying to call python split() function on the whole list of lines, and you can’t split a list of strings, only a string. So, you need to split each line, not the whole thing.

To solve the above problem, you need to iterate over the strings in the list to get individual strings; then, you can call the split() function.