I saw the tutorial on custom dataloader. However, the class function has loading data functions too. I have tensors pair images, labels. How can I convert them into DataLoader format without using CustomDataset class??
I am getting this error:
error: invalid cast to abstract class type ‘CustomDataset’
auto data = CustomDataset(train_tensor_indices, train_tensor_lab).map(torch::data::transforms::Stack<>());
note: because the following virtual functions are pure within ‘CustomDataset’:
class CustomDataset : public torch::data::Dataset
^
In file included from /libtorch/include/torch/csrc/api/include/torch/data/datasets.h:3:0,
from /libtorch/include/torch/csrc/api/include/torch/data.h:4,
from /libtorch/include/torch/csrc/api/include/torch/all.h:4,
from /libtorch/include/torch/csrc/api/include/torch/torch.h:3,
Custom Dataset class:
class CustomDataset : public torch::data::Dataset {
private:
torch::Tensor states_, labels_;
public:
CustomDataset(torch::Tensor emb_indices, torch::Tensor labels) {
states_ = emb_indices;
labels = labels;
};
torch::data::Example<> get(size_t index) override {
torch::Tensor embedding_index = states_[index];
torch::Tensor Label = labels_[index];
return {embedding_index.clone(), Label.clone()};
}; };
auto data = CustomDataset(train_tensor_indices, train_tensor_lab).map(torch::data::transforms::Stack<>());
@avinash_m
I think you need to implement a size() as well. See here:
‘https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/include/torch/data/datasets/base.h#L53’
Dataset inherited from BatchDataset and it has a pure virtual function size() which is not overridden.
As the comment said, if you don’t need it, just return an empty c10::optional
Thank you for the suggestion. I will try and let you know.
@glaringlee Thanks for the suggestion. It worked. I need one more clarification.though not in scope to this question. The loss functions doesn’t support Long Tensors for backward pass?
@avinash_m
I need more details on ‘loss’ question.
Do you have any log or error msg? I think we should support Long Tensors (the tensor’s datatype is Long correct? correct me if I am wrong)
Code:
for(auto& batch : trainloader) {
auto inputs = at::_cast_Long(batch.data).to(device);
auto targets = at::_cast_Long(batch.target).to(device);
totalsize = totalsize + inputs.size(0);
optimizer.zero_grad();
auto output = Net.forward(inputs);
auto loss = torch::nn::functional::cross_entropy(inputs, targets);
loss.backward();
optimizer.step();
Error Message:
terminate called after throwing an instance of ‘c10::Error’
what(): “log_softmax_lastdim_kernel_impl” not implemented for ‘Long’ (operator() at aten/src/ATen/native/cpu/SoftMaxKernel.cpp.AVX2.cpp:238)
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0x6a (0x7f6e629dcaaa in /libtorch/lib/libc10.so)
frame #1: + 0x154c8ff (0x7f6e54fe58ff in /libtorch/lib/libtorch_cpu.so)
frame #2: at::native::log_softmax_cpu(at::Tensor const&, long, bool) + 0x54a (0x7f6e548ecc1a in /libtorch/lib/libtorch_cpu.so)
frame #3: + 0x115ef07 (0x7f6e54bf7f07 in /libtorch/lib/libtorch_cpu.so)
frame #4: + 0x11b16ab (0x7f6e54c4a6ab in /libtorch/lib/libtorch_cpu.so)
frame #5: + 0x2d497a2 (0x7f6e567e27a2 in /libtorch/lib/libtorch_cpu.so)
frame #6: + 0x11b16ab (0x7f6e54c4a6ab in /libtorch/lib/libtorch_cpu.so)
frame #7: at::native::log_softmax(at::Tensor const&, long, c10::optionalc10::ScalarType) + 0x160 (0x7f6e548ea0e0 in /libtorch/lib/libtorch_cpu.so)
frame #8: + 0x1227333 (0x7f6e54cc0333 in /libtorch/lib/libtorch_cpu.so)
frame #9: + 0x2ce3cf5 (0x7f6e5677ccf5 in /libtorch/lib/libtorch_cpu.so)
frame #10: + 0x12987cc (0x7f6e54d317cc in /libtorch/lib/libtorch_cpu.so)
yes, data is long type.
@avinash_m
I see. log_softmax_lastdim_kernel_impl is implemented for floating types. Specifically, it supports double, float and torch::Half only. You might need to use double instead of long.
@glaringlee. But Embedding layer only takes Long Tensors as input. I have to use Embedding layers. Is there any work around to avoid this issue? Should I raise an issue in the official github repo?
@avinash_m
Can you post more code?
Based on the code piece you post here, I think you can cast your inputs and targets to double first. If you have other layers that need long, you can cast them to long later.
Code:
struct RandCNN : nn::Module {
RandCNN(int32_t &vocabsize, int32_t &embeddim, int32_t &maxlen, int32_t &numclasses)
: embed(nn::Embedding(nn::EmbeddingOptions(vocabsize, embeddim))),
conv1(nn::Conv1d(nn::Conv1dOptions(embeddim, kNumfilters, kFilterSizes[0])
.stride(1)
.bias(false))),
conv2(nn::Conv1d(nn::Conv1dOptions(embeddim, kNumfilters, kFilterSizes[1])
.stride(1)
.bias(false))),
conv3(nn::Conv1d(nn::Conv1dOptions(embeddim, kNumfilters, kFilterSizes[2])
.stride(1)
.bias(false))),
pool1(nn::MaxPool1d(nn::MaxPool1dOptions(maxlen - kFilterSizes[0] +1)
.stride(1))),
pool2(nn::MaxPool1d(nn::MaxPool1dOptions(maxlen - kFilterSizes[1] +1)
.stride(1))),
pool3(nn::MaxPool1d(nn::MaxPool1dOptions(maxlen - kFilterSizes[2] +1)
.stride(1))),
fc(nn::Linear(nn::LinearOptions(kNumfilters * 3, numclasses))),
drop(nn::Dropout(nn::DropoutOptions().p(kDropValue))) {
// register_module() is needed if we want to use the parameters() method later on
register_module("embed", embed);
register_module("conv1", conv1);
register_module("conv2", conv2);
register_module("conv3", conv3);
register_module("poo11", pool1);
register_module("pool2", pool2);
register_module("pool3", pool3);
register_module("fc", fc);
}
torch::Tensor forward(torch::Tensor x) {
x = embed(x);
x = at::transpose(x, 1, 2);
torch::Tensor out1, out2, out3, out;
out1 = torch::relu(conv1(x));
out2 = torch::relu(conv2(x));
out3 = torch::relu(conv3(x));
out1 = pool1(out1);
out2 = pool2(out2);
out3 = pool3(out3);
out = at::_cat({out1, out2, out3}, 1);
out = at::_unsafe_view(out, {at::size(out, 0), at::size(out, 1)});
out = drop(out);
out = fc(out);
//out = torch::log_softmax(out, /*dim=*/1);
return out;
}
My dataformat will be a dataloader with Indices and Labels. The first layer of model Embedding converts them to embedding vectors. So, inputs will be Long Tensors.
I see.
In your forward function, when calling the log_softmax, you can do the conversion
out = torch::log_softmax(out.to(torch::kDouble)).to(torch::kLong) if you need Long type as output
okay so i will use long type as input to embedding layer. then type cast to double after this so that it will be backward compatible? But the problem is back-propogation thorugh long tensors is giving error.
@avinash_m
A typo in my previous reply, it should be out = torch::log_softmax(out.to(torch::kDouble), 1).to(torch::kLong)
It should be backward compatible. Can I see your back-propogation error log?
The previous error log you showed me is caused by calling log_softmax in forward.
@glaringlee I have identified the issue after careful debugging. In this line I should be passing outputs and targets to calculate loss however I was passing inputs. However, the error log is not useful to trace back the error. It should have reported shapes mismatch instead was reporting about log_softmax.
Thank you for all the patience and time in helping me throughout.
.I guess there should have been a better error log. Because clearly it is a shape mismatch.