Transformer not able to do bracket matching

[yamsk]

I have been trying to apply transformers to classify simpler sequences. One such example is in my earlier post with working code:

Transformer layer weights not updating

Hi: I am trying to classify simple sequence data. Examples: ([‘bbRcabbSab’, ‘abRbbccSca’, ‘baabRcaaaS’, ‘abRbabaSaa’, ‘acRcbacSbc’, ‘bcRbaacSbc’, ‘RbbaaabaSc’, ‘abRacabSba’, ‘RabcbSbcba’, ‘ccRbcaaSba’], [1, 1, 0, 1, 1, 1, 0, 1, 0, 1]) If R is in position 2 and S is in position len(s)-3 then the target is 1 and 0 otherwise. I am trying to use a transformer for this. The model I am using is: class PositionalEncoding(nn.Module): def __init__(self, dim_model, dropout_p, max_len=5…

However, the code does not work for the following bracket matching examples. I was told that matching brackets is quite a tricky problem for transformers as it has high circuit complexity. But if someone wants to take a crack at it. Or try it for any other simpler sequences.

Bracket matching examples.
([‘()()()()()’, ‘()()))()()’, ‘())()())()’, ‘(((())))))’, ‘()()))()()’],
[1, 0, 0, 0, 0])