Defining custom internal activation functions on LSTM

1

Hi, I’m trying to implement custom internal activation function for LSTM. Essentially I’m trying to replace the LSTM cell’s tanh function with torch.sin. I’m very new to machine learning and PyTorch, and from what I’ve understood you can implement custom activation functions between layers, but is there any way to use custom function in the cells themselves?

2

You can create custom activation functions in PyTorch and use them in your LSTM cells. To replace the tanh activation function in LSTM cells with your custom function (e.g., torch.sin), you’ll need to modify the LSTM cell implementation.The LSTM cell in PyTorch has default activations: activation=“tanh” and recurrent_activation=“sigmoid”.

import torch
import torch.nn as nn

class CustomLSTMCell(nn.Module):
def init(self, input_size, hidden_size):
super().init()
self.input_size = input_size
self.hidden_size = hidden_size
self.weight_ih = nn.Parameter(torch.Tensor(4 * hidden_size, input_size))
self.weight_hh = nn.Parameter(torch.Tensor(4 * hidden_size, hidden_size))
self.bias_ih = nn.Parameter(torch.Tensor(4 * hidden_size))
self.bias_hh = nn.Parameter(torch.Tensor(4 * hidden_size))
# Initialize weights and biases here

def forward(self, input, hx):
    # Compute LSTM cell operations using custom activation functions
    gates = torch.matmul(input, self.weight_ih.t()) + self.bias_ih + torch.matmul(hx, self.weight_hh.t()) + self.bias_hh
    ingate, forgetgate, cellgate, outgate = gates.chunk(4, 1)
    cellgate = torch.sin(cellgate)  # Custom activation (replace with torch.sin)
    cy = (forgetgate * hx) + (ingate * cellgate)
    hy = torch.tanh(cy)  # Custom activation (replace with torch.tanh)
    return hy, cy