Basic Extreme Learning Machine pytorch implementation

derby · April 9, 2024, 7:56pm

Hi all, I wanted to share this little github gist i made that is a basic implementation of the ELM, “extreme learning machine”. a quick explainer is that its a single hidden layer FFNN with randomized and locked/frozen hidden neurons. the input weights/biases are not computed using backprop; it uses least squares solutions and psuedo-inverse of the hidden layer outputs multiplied by target outputs analytically. there are many papers on the ELM and will describe it much better than me.

in this demonstration we classify the MNIST dataset, and our results are seemingly quite good! here are the results of a few runs (data preprocessing takes much longer than the actually training):

1000 Hidden Neurons:
Training Time: 2.3480 seconds
Prediction Time: 0.0872 seconds
Test Accuracy: 94.12%

5000 Hidden Neurons:
Training Time: 39.7465 seconds
Prediction Time: 0.3457 seconds
Test Accuracy: 97.28%

10000 Hidden Neurons:
Training Time: 213.8608 seconds
Prediction Time: 0.6887 seconds
Test Accuracy: 97.66%

maybe too good to be true? idk, see for yourself:

gist.github.com

https://gist.github.com/derbydefi/616b1d5b986610f7cb5120ccaac85915

elm.py

import time
import torch
import torchvision
import torchvision.transforms as transforms

class ELM(torch.nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(ELM, self).__init__()
        self.hidden_size = hidden_size
        self.input_weights = torch.nn.Parameter(torch.randn(input_size, hidden_size) * 0.1, requires_grad=False)

This file has been truncated. show original