I am testing a two-step architecture that is composed of a conventional first section that can be implemented with any standard deep learning architecture and a second section that must be coded manually outside the declaration of the Pytorch graph (while still utilizing numpy-like torch functions).
My problem can be simplified to coding a feed-forward neural network with two hidden layers, where the first is implemented within the Pytorch graph and the second is implemented manually outside the Pytorch graph.
Architecture:
-> Linear(28 * 28, 120) in Pytorch graph
-> ReLU in Pytorch graph
-> Linear(120, 84) in Pytorch graph
-> ReLU in Pytorch graph
-> Linear(84, 10) outside of Pytorch graph
-> Output
Problem: My implementation below achieves a very low ~74%, while a standard fully Pytorch implementation achieves ~95%. What is causing this disparity?
I believe my problem lies in manually passing back the deltas, although the math looks right, so I am stuck in finding a solution to this.
Implementation of architecture and training on MNIST:
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.fc1 = nn.Linear(28 * 28, 120)
self.fc2 = nn.Linear(120, 84)
def forward(self, x):
x = x.view(-1, 28 * 28)
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
return x
net = Net()
criterion = nn.MSELoss()
optimizer = optim.SGD(net.parameters(), lr=0.01)
# Initialize weights just as Pytorch does by default:
m = torch.distributions.uniform.Uniform(torch.tensor([-np.sqrt(1.0/84)]),
torch.tensor([np.sqrt(1.0/84)]))
W = m.sample((84, 10)).reshape((84, 10))
# based on https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html
for epoch in range(2): # loop over the dataset multiple times
for i, data in enumerate(trainloader, 0):
# get the inputs
inputs, labels = data
# make one-hot encoding of labels
targets = oneHot(labels)
# zero the parameter gradients
optimizer.zero_grad()
# forward + backward + optimize
pytorch_outputs = net(inputs)
pytorch_outputs = torch.autograd.Variable(pytorch_outputs,
requires_grad=True)
manual_outputs = torch.mm(pytorch_outputs, W)
delta_out = manual_outputs - targets.view(-1,10) # = error_out
dEdW3 = torch.mm(torch.t(pytorch_outputs), delta_out)
W -= 0.01 * dEdW3 # gradient descent
delta_h = torch.autograd.Variable(
torch.t(torch.mm(W, torch.t(delta_out))))
loss = criterion(pytorch_outputs, delta_h)
loss.backward()
optimizer.step()
Full code: https://pastebin.com/EM5q4P6w