I cannot reproduce the issue and see gradients in your new custom layers:
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
out = model(**inputs)
loss = out[0].mean()
loss.backward()
print(model.transformer.h[0].attn.c_proj.adapter.adapter_block[0].weight.grad.abs().sum())
# tensor(6387.1465)
I also had to fix a previous error in your code snippet raising:
# TypeError: dropout(): argument 'input' (position 1) must be Tensor, not tuple
by removing the tuple creation in your custom module.