I’m building big machine learning project. In this project I have many different models, with different inputs, outputs, and serving different purposes. I like to keep code transparent, simple and flexible, so I use various design patterns (loads of Strategy design patterns, Dependency Injections and so one).
But now comes my question: Do you guys have idea how to design save, load system? For example I have one model:
class AutoEncoder(nn.Module): def __init__(self, input_size, encoded_size, ac_function='relu'): super().__init__() self.ac_function = ac_function self.encoded_size = encoded_size self.input_size = input_size ...
And I want to save it and keep information about input_size, encoded_size and so one…
Also each model uses some sort of Normalizations, Standardizations to transform dataset data, like:
class Standardization(StatisticTransformation): def __init__(self, mean, std): self.mean = mean self.std = std def _apply_key_val(self, key, val, indexes=None):
So with each model I’d like to keep information about type of Transformation and for example means and stds.
I’d love to discuss with someone about it.