Multi-Task Classifier

nlp
1

I am working on a Multi-Task text classifier, I want to make it config based i.e. I have an x input but not fix set of tasks. tasks are decided based on the config.
So how can I initialize different Classifiers based on config?

eg:- if my config has 2 tasks then 2 linear layers are initialized in model init and if my config has 5 tasks then 5 linear layers are initialized.

2

You could create a config file as e.g. a JSON file and read it in your script via pandas.
Once you have the right keys from your config, you could use nn.ModuleList or nn.Sequential to create the number of layers based on the current config.

The ModuleList docs give you an example:

class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        self.linears = nn.ModuleList([nn.Linear(10, 10) for i in range(10)])

    def forward(self, x):
        # ModuleList can act as an iterable, or be indexed using ints
        for i, l in enumerate(self.linears):
            x = self.linears[i // 2](x) + l(x)
        return x

Instead of the fixed 10 you could use the value passed in your config file.

4

I am adding this layer on top of hugginging face transformers layers, and in params it is showing that these layers are added, training loss is reducing greatly but eval loss is very high and increasing.