Hello,
I think I have some issues with loading textdata. Since I want to create a text generator the input for my network is a string of 30 chars. With the following two lines I want to load the data.
# train_text is the string I want the network to learn
# args.seq_len is the number of chars the input strings should have
train_set = TextDataset(train_text, args.seq_len, args.offset)
train_loader = DataLoader(train_set, batch_size=args.batch_size, shuffle=True)
The train_set includes the strings like this (the text is german):
(0, ([' ', 'w', 'รค', 'h', 'r', 'e', 'n', 'd', ' ', 'd', 'i', 'e', ' ', 'h', 'i', 's', 't', 'o', 'r', 'i', 's', 'c', 'h', 'e', ' ', 'z', 'a', 'h', 'n', 'r'], 'a'))
The chars in the list are regular words and the char after the list is the next that the network should output.
The train_loader includes:
# length is the batch size. One batch have 30 columns what is the size of the string length for the input
(0, [[(' ', 'r', ' ', ' ', 't', 's', ' ', 'n', 'b', ' ', 'h', 'h', ' ', 'e', 'e', 'e', 'c', 'n', 'e', 'g', ' ', 'i', ' ', 'g', 'n', 'k', 'l', 'b', 'k', 'e', 'n', 'i', 'h', 'f', 'e', 'z', 't', 'b', 's', 'z', 'h', ' ', 'r', 'm', ' ', ' ', 'n', 'i', 'n', 'a', 'i', 'i', 'l', 'e', 'n', 's', 'r', 't', 'f', 'e', 'n', 'i', ' ', 'i', 'd', 'w', 'l', 'i', 'k', 't', 'e', 'd', 'b', 'd', 'a', 'u', 'z', 'c', 'b', 'a', 'h'),.....
In the train_loader the words are saved in the columns, not in the rows (index 0 of all 30 rows are the input string). I guess that is wrong. I hope you can understand what I mean and someone can help me.