Custom mnist-like dataset

C++
1

Hello!
I want to train LeNet in libtorch with using c++ api.
I created my own train-images-idx3-ubyte and idx1 from printed digits.
I load it as following:

    auto train_data_set = torch::data::datasets::MNIST(mnist_dataset_path, torch::data::datasets::MNIST::Mode::kTrain)
                              .map(torch::data::transforms::Normalize<>(0.5, 0.5))
                              .map(torch::data::transforms::Stack<>());

Here, when i try to load it, it says:

terminate called after throwing an instance of 'c10::Error'
  what():  Expected to read number 60000 but found 1930 instead
Exception raised from expect_int32 at /root/pytorch/torch/csrc/api/src/data/datasets/mnist.cpp:49 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6c (0x7c49350c29ac in /lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xfa (0x7c4935089008 in /lib/libc10.so)
frame #2: <unknown function> + 0x4e13a99 (0x7c4931a13a99 in /lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x4e13e82 (0x7c4931a13e82 in /lib/libtorch_cpu.so)
frame #4: torch::data::datasets::MNIST::MNIST(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, torch::data::datasets::MNIST::Mode) + 0x4a (0x7c4931a159ca in /lib/libtorch_cpu.so)
frame #5: main + 0x54a (0x5b947b41f2f9 in /home/dmitriys/projects/ray/MNIST-cls-cpp/build/train)
frame #6: <unknown function> + 0x2a1ca (0x7c492c42a1ca in /lib/x86_64-linux-gnu/libc.so.6)
frame #7: __libc_start_main + 0x8b (0x7c492c42a28b in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: _start + 0x25 (0x5b947b41ea95 in /home/dmitriys/projects/ray/MNIST-cls-cpp/build/train)

How can I use my own dataset which has 28x28 images (printed digits), MNIST like?

2

The predefined MNIST dataset class in libtorch uses the full MNIST dataset, i.e. 60000 samples in training mode, which is hard-coded in the libtorch library. I think Libtorch just provides a reference example to allow CPP programmers to define their own datasets according to their own problems. https://github.com/pytorch/pytorch/blob/main/torch/csrc/api/src/data/datasets/mnist.cpp
If you want to use your own custom dataset, I think the best way is to define a class similar to the MNIST class. Or read the file directly and convert the data into tensors, then use the data loader and blah, blah。