The conv layers cant magically handle nan values using some kind of interpolation/semi-supervised learning/etc. So you need yourself to figure out how to convert the nan values into an actual concrete value that will make sense to the model you have written. As Patrick says, using the median/average value might be a quick way to get something working. Otherwise you can look at more complex techniques, which you could find by googling eg ‘handling missing data’, which throws up eg https://en.wikipedia.org/wiki/Missing_data (I imagine a ton of Kaggle competition write-ups would give some insight into how to handle missing data ?)