Hi! I have a question about how to best plot things. Suppose I have a model and that during inference I am getting the error per example in a batch. I then want to correlate the error of the predicted y with a property of the input x. So ultimately I’m plotting a huge amount of property(x) and error(y,y_pred) on a scatterplot, and then calculating their correlation. I assume that the best way to do this would be to log every batch x properties and y errors using a custom metric, and then outputing the correlation between the property(x) and error(y,y_pred) for all x,y pairs at the end through the compute() function.
However, what if I also wanted a scatterplot of all the property(x) and error(y,y_pred) for all pairs x,y? I understand that metric stores the history, but would it be too convoluted to also plot all the property(x) and error(y,y_pred) from within Metric’s compute() function? It seems complicated because I create my visdom Visualizer in a file called “train.py” (where I also run my evaluator) and the custom Metric is in its own file called metric.py.
The only other option I can think of is to store all the property(x) and error(y,y_pred) outside of the Metric.py file by returning the property(x) and error(y,y_pred) in the dict returned by the evaluator engine in train.py, and then creating an event handler that will save those values every iteration until the end of the validation set. After this, all the property(x) and error(y,y_pred) are saved in a file, and I just repull them all to do a scatter plot.
I was wondering what is the best/more ignite/most elegant way to do this.