Ignore zero in weights of model for inference

I have pruned a model and after pruning the model has lot of zeros in layers. After doing inference there is no improvement in speed, So is there any way to ignore zeros preset in layers and achieve after inference time.
Just to give more ideas here are some layers info

electra.embeddings.word_embeddings.weight | nonzeros = 19355528 / 23887104 ( 81.03%) | total_pruned = 4531576 | shape = (31103, 768)
electra.embeddings.position_embeddings.weight | nonzeros =  177670 /  393216 ( 45.18%) | total_pruned =  215546 | shape = (512, 768)
electra.embeddings.token_type_embeddings.weight | nonzeros =     132 /    1536 (  8.59%) | total_pruned =    1404 | shape = (2, 768)
electra.embeddings.LayerNorm.weight | nonzeros =     768 /     768 (100.00%) | total_pruned =       0 | shape = (768,)

I’d greatly appreciate any kind of feedback on this.
Thank you all in advance