Is there a way to save a model from a running script, when not mentioned in the code? I am running a model since 5 days on GPU and save the model whenever a new highscore in the metric is achieved. Unfortunately, the dataset contains many mislabeled samples. Thus, the score is declining at the moment although from a qualitative perspective the model is still making progress. As a result, the model is not saved anymore. Is there a way to save the current model or do I have to run it again?
I don’t think this is possible in a clean way. However, you could check, if you can find any usable code injection libraries, which might be able to inject the saving code into the running process.
Note that this workflow is quite hacky, but you might be able to avoid rerunning the entire training.