Feature-Level Fusion

1

Hi everyone,

I am working on a mango ripeness classification project using PyTorch.

My architecture is:

  • Arsitektur Hybrid MobileNetV2 + HSV
                    +----------------------+
                    |   Input RGB Image    |
                    +----------+-----------+
                               |
               +---------------+---------------+
               |                               |
               v                               v
    +---------------------+       +----------------------+
    |     MobileNetV2     |       |   HSV Extraction     |
    | Feature Extractor   |       | (Mean & Std H,S,V)   |
    +----------+----------+       +----------+-----------+
               |                             |
               v                             v
    +---------------------+       +----------------------+
    | 1280-D CNN Feature  |       |   6 HSV Features     |
    +----------+----------+       +----------+-----------+
               |                             |
               |                             v
               |                 +----------------------+
               |                 |      HSV MLP         |
               |                 | Linear(6→32)         |
               |                 | ReLU                 |
               |                 | Linear(32→16)        |
               |                 +----------+-----------+
               |                             |
               +-------------+---------------+
                             |
                             v
                 +--------------------------+
                 |      Concatenation        |
                 |      1280 + 16 = 1296     |
                 +-------------+------------+
                               |
                               v
                 +--------------------------+
                 |      Classifier Head      |
                 |     Linear(1296→256)      |
                 |          ReLU             |
                 |      Linear(256→3)        |
                 +-------------+------------+
                               |
                               v
                 +--------------------------+
                 |   Output (3 Classes)     |
                 +--------------------------+

Description:

  • The left branch uses MobileNetV2 to extract high-level visual features (1280 features).
  • The right branch extracts 6 HSV color features (mean and standard deviation of H, S, and V), which are then processed by MLP to produce a 16-dimensional representation.
  • The two features are concatenate into a 1296-dimensional vector.
  • The combined vector is classified using a fully connected classifier into three mango ripeness classes.

My question:

Would this architecture be considered a valid hybrid CNN + handcrafted feature fusion approach?

Or would it be more appropriate to convert the image to HSV and feed it into a second CNN branch (dual-stream architecture) before feature fusion?

I am trying to understand whether concatenating MobileNetV2 deep features with manually extracted HSV statistics is a sound methodology from a deep learning perspective.

Any feedback or references would be greatly appreciated.

2

Hello! Even if it is true that handcrafted features can speed up training or make it even possible when using DL, in your case, I doubt bout the usefulness of the rigth branch. I’m not an expert but here are my thouths:

  • You have the hippotesis that HSV aggregated values of an image are useful. That may be true. In general it wont help much because ‘shape’ and ‘texture’ are not captured, just ‘color’ but the same, say, orange, under different ligth may have different HSV values. SO, if that hippostesis is true, then that relationship must be captured somehow in your mobilenet, for sure. That meaning that the addition of hand crafted features shall not improve model’s expresiveness.

Hope it helps

have a nice day

3

Dropping the MLP and concatenating the raw 6-dimensional HSV statistics directly with the 1280-dimensional MobileNetV2 output (yielding a 1286-dimensional vector) is a cleaner, more minimalist design choice. Yet, your proposed architecture is a fully valid and methodologically sound hybrid CNN + handcrafted feature fusion approach.