Very low accuracy when fine-tuning DeGCN pretrained on NTU RGB+D to UCF101 skeleton data

1

Hello,

I am working on skeleton-based action recognition with DeGCN and I am facing a transfer learning issue.

First, I trained DeGCN on the NTU RGB+D dataset and obtained a pretrained model. Then I tried to fine-tune this model on skeleton data extracted from UCF101. However, the performance on the target dataset remains quite low.

To make the source and target data more compatible, I mapped both datasets into a 15-joint format. I checked the mapping logic carefully, but the transfer result is still weak.

My setup is as follows:

  • Source dataset: NTU RGB+D

  • Target dataset: UCF101 skeleton data

  • Model: DeGCN

  • Fine-tuning strategy: continuing from pretrained weights trained on NTU

  • Joint setting: both datasets converted to 15 joints

  • Input tensor format: [N, C, T, V, M]

  • UCF101 skeleton type: [2D or 3D]

  • Pose extractor for UCF101: [write here]

What I observe is:

  • training runs normally,

  • but the accuracy on the target dataset stays very low,

  • and using pretrained weights does not seem to provide a clear benefit.

At this point, what I would like to ask is:

Is it normal for transfer learning from NTU RGB+D to UCF101 to perform poorly in skeleton-based action recognition?