I’m working on an classification problem with images at the pixel level using pytorch. The images are all 10,000 pixels wide and high. To elaborate there are two kinds of pixels in this image “x” pixels and “y” pixels. It is impossible to tell them apart in a single image. However, the pixels for “x” change in a very specific way over time and the idea is to identify “x” as soon as possible. I have a stack of images (timestep of 50 images) involving these pixels. Additionally, “x” pixels are about 25% of the image while “y” pixels are the remaining 75%. The final image I have is a black and white image in which constitutes the labels named key.jpg. Basically “x” is black and “y” is white this is what is used to identify “x” and “y” pixels over time. The idea is to have the nn determine if a pixel is an “x” or a “y” pixel by the 5th timestep, that way I can use the nn on a new series of images that have “x” and “y” pixels in a different configuration have it classify them enabling the creation of a matching key.jpg for them.
I theorize that I can solve this with a cnn or an rnn (lstm), but I’m not sure of how to implement them. The general idea I have stack the images above one another in the order of time (an array) then select 500 random pixels following them over time with 25% of them being “x” and 75% of them being “y”. Use 200 pixels as training, 200 as validating and 100 for testing (while keeping the pixel ratios).
The problem I have here is that I cannot envision what my next step should be like and I’m facing two problems.
-
I’m clearly working with time series data so a cnn might not be the right choice.
-
I cannot figure out how to use this in an rnn. Logically I can envision using a single time series, but since each pixel’s tuple/array is an individual time series and I’m not sure how to proceed. Additionally, I’m not sure how I’d infer the timesteps in pytorch
Any help/suggestions would be greatly appreciated.