训练稀疏自动编码器 [英] Training Sparse Autoencoders

问题描述

我的数据集由大量矢量组成.数据点几乎都是零，约有3％的特征是1.本质上，我的数据非常稀疏，我正在尝试训练自动编码器，但是我的模型只是在学习重新创建所有零的向量.

My dataset consists of vectors that are massive. The data points are all mostly zeros with ~3% of the features being 1. Essentially my data is super sparse and I am attempting to train an autoencoder however my model is learning just to recreate vectors of all zeros.

有什么方法可以防止这种情况发生?我曾尝试用骰子损失代替均方误差，但是它完全停止了学习.我的其他想法是使用损失函数，该函数有助于正确猜测1，而不是0.我也尝试过使用S型和线性最后一次激活，但没有明确的获胜者.任何想法都很棒.

Are there any techniques to prevent this? I have tried replacing mean squared error with dice loss but it completely stopped learning. My other thoughts would be to use a loss function that favors guessing 1s correctly rather than zeros. I have also tried using a sigmoid and linear last activation with no clear winner. Any ideas would be awesome.

推荐答案

似乎您正面临严重的阶级失衡".问题.

It seems like you are facing a severe "class imbalance" problem.

1. 查看焦点损失.这种损失是针对具有严重的类不平衡的二进制分类而设计的.

1. Have a look at focal loss. This loss is designed for binary classification with severe class imbalance.

考虑硬否定挖掘":即，仅针对部分训练示例传播梯度-硬"挖掘一个.
参见，例如:
Abhinav Shrivastava，Abhinav Gupta和Ross Girshick 通过在线培训基于区域的对象检测器困难示例挖掘 (CVPR 2016).

Consider "hard negative mining": that is, propagate gradients only for part of the training examples - the "hard" ones.
see, e.g.:
Abhinav Shrivastava, Abhinav Gupta and Ross Girshick Training Region-based Object Detectors with Online Hard Example Mining (CVPR 2016).

这篇关于训练稀疏自动编码器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！