LSTM自动编码器问题 [英] LSTM Autoencoder problems
问题描述
TLDR:
自动编码器不适合时间序列重建，只能预测平均值.
问题设置:
这是我尝试使用序列到序列自动编码器的摘要.此图像摘自本文:
编码器:标准LSTM层.输入序列被编码为最终的隐藏状态.
解码器: LSTM单元(我认为！).从最后一个元素 x [N] 开始，一次重构一个元素.
N 的序列，解码器算法如下:
- 获取解码器初始隐藏状态
hs [N]:只需使用编码器最终隐藏状态即可. - 重建序列中的最后一个元素:
x [N] = w.dot(hs [N])+ b. - 其他元素的相同模式:
x [i] = w.dot(hs [i])+ b - 使用
x [i]和hs [i]作为LSTMCell的输入以获得x [i-1]和hs [i-1]
最小工作示例:
这是我的实现，从编码器开始:
SeqEncoderLSTM(nn.Module)类:def __init __(self，n_features，latent_size):超级(SeqEncoderLSTM，self).__ init __()self.lstm = nn.LSTM(n_features，latent_size，batch_first = True)def forward(self，x):_，hs = self.lstm(x)返回hs 解码器类:
SeqDecoderLSTM(nn.Module)类:def __init __(self，emb_size，n_features):超级(SeqDecoderLSTM，self).__ init __()self.cell = nn.LSTMCell(n_features，emb_size)self.dense = nn.Linear(emb_size，n_features)def forward(自我，hs_0，seq_len):x = torch.tensor([])#编码器的最终隐藏状态和单元状态hs_i，cs_i = hs_0#用编码器输出重建第一个元素x_i = self.dense(hs_i)x = torch.cat([x，x_i])#重建剩余元素对于范围(1，seq_len)中的i:hs_i，cs_i = self.cell(x_i，(hs_i，cs_i))x_i = self.dense(hs_i)x = torch.cat([x，x_i])返回x 将两者结合在一起:
LSTMEncoderDecoder类(nn.Module):def __init __(self，n_features，emb_size):超级(LSTMEncoderDecoder，self).__ init __()self.n_features = n_featuresself.hidden_size = emb_sizeself.encoder = SeqEncoderLSTM(n_features，emb_size)self.decoder = SeqDecoderLSTM(emb_size，n_features)def forward(self，x):seq_len = x.shape [1]hs = self.encoder(x)hs =元组([h中以hs为单位的[h.squeeze(0)])out = self.decoder(hs，seq_len)返回out.unsqueeze(0) 这是我的训练功能:
def train_encoder(模型，历元，训练量，测试量=无，标准= nn.MSELoss()，优化程序= optim.Adam，lr = 1e-6，reverse = False):设备='cuda'如果torch.cuda.is_available()否则为'cpu'打印(f"{设备}上的训练模型")型号= model.to(设备)选择=优化程序(model.parameters()，lr)train_loss = []valid_loss = []对于tqdm中的e(range(epochs)):running_tl = 0running_vl = 0对于火车中的x:x = x.to(设备).float()opt.zero_grad()x_hat =模型(x)如果相反:x = torch.flip(x，[1])损失=准则(x_hat，x)loss.backward()opt.step()running_tl + = loss.item()如果testload不为None:model.eval()使用torch.no_grad():对于testload中的x:x = x.to(设备).float()损失=准则(model(x)，x)running_vl + = loss.item()valid_loss.append(running_vl/len(testload))model.train()train_loss.append(running_tl/len(火车载荷))返回train_loss，valid_loss 数据:
从新闻(ICEWS)抓取的事件的大型数据集.存在描述每个事件的各种类别.最初，我对这些变量进行了一次热编码，将数据扩展到274个维度.但是，为了调试模型，我将其缩减为一个序列，该序列长14个时间步长，并且仅包含5个变量.这是我尝试过度拟合的顺序:
张量([[0.5122，0.0360，0.7027，0.0721，0.1892]，[0.5177，0.0833，0.6574，0.1204，0.1389]，[0.4643、0.0364、0.6242、0.1576、0.1818]，[0.4375、0.0133、0.5733、0.1867、0.2267]，[0.4838，0.0625，0.6042，0.1771，0.1562]，[0.4804、0.0175、0.6798、0.1053、0.1974]，[0.5030、0.0445、0.6712、0.1438、0.1404]，[0.4987、0.0490、0.6699、0.1536、0.1275]，[0.4898、0.0388、0.6704、0.1330、0.1579]，[0.4711、0.0390、0.5877、0.1532、0.2201]，[0.4627，0.0484，0.5269，0.1882，0.2366]，[0.5043，0.0807，0.6646，0.1429，0.1118]，[0.4852、0.0606、0.6364、0.1515、0.1515]，[0.5279，0.0629，0.6886，0.1514，0.0971]]，dtype = torch.float64) 这是自定义的 Dataset 类:
class TimeseriesDataSet(Dataset):def __init __(自身，数据，窗口，n_features，overlap = 0):super().__ init __()如果isinstance(data，(np.ndarray)):数据= torch.tensor(数据)elif isinstance(数据，(pd.Series，pd.DataFrame)):数据= torch.tensor(data.copy().to_numpy())别的:引发TypeError(f数据应为ndarray，系列或数据框.找到的{type(data)}."")self.n_features = n_featuresself.seqs = torch.split(数据，窗口)def __len __(自己):返回len(self.seqs)def __getitem __(self，idx):尝试:返回self.seqs [idx] .view(-1，self.n_features)除了TypeError:引发TypeError(数据集仅接受整数索引/切片，不接受列表/数组.") 问题:
无论我制作模型有多复杂，或者现在训练它多长时间，模型都只会学习平均值.
预测/重建:
实际:
我的研究:
此问题与该问题中讨论的问题相同:
已达到1000次迭代限制
SUBTRACT，小模型
-
HIDDEN_SIZE = 5 -
SUBTRACT = True
目标现在已经远离平坦线，但是由于容量太小而无法拟合模型.
已达到1000次迭代限制
无减法，更大型号
-
HIDDEN_SIZE = 100 -
SUBTRACT = False
情况变得更好了，我们的目标是在 942 步骤之后达到的.没有更多的扁平线，模型的容量似乎还不错(对于这个例子！)
SUBTRACT，更大的模型
-
HIDDEN_SIZE = 100 -
SUBTRACT = True
尽管该图看起来并不漂亮，但仅经过 215 次迭代，我们就达到了预期的损失.
最后
- 通常使用时间步长而不是时间步长(或其他一些转换，请参见此处以获取有关此信息的更多信息).在其他情况下，神经网络将尝试简单地...复制上一步的输出(因为这是最简单的操作).通过这种方式可以找到一些最小值，而超出最小值将需要更多的容量.
- 当您使用时间步长之间的差异时，将无法推算"出结果.先前时间步长的趋势；神经网络必须了解功能的实际变化
- 使用更大的模型(对于整个数据集，您应该尝试使用
300之类的方法)，但是您可以简单地对其进行调整. - 请勿使用
flipud.使用双向LSTM，通过这种方式，您可以从LSTM的向前和向后传递中获取信息(不要与反向传播混淆！).这也应该提高您的分数
问题
好吧，问题1:您说的是时间中的变量x系列，我应该训练模型以学习x [i]-x [i-1]而不是x [i]的值?我的解释正确吗?
是的，完全正确.差异消除了神经网络过多地将其预测基于过去时间步长的冲动(通过简单地获取最后一个值并可能对其进行了一些更改)
问题2:您说我对零瓶颈的计算是不正确.但是，例如，假设我使用的是简单的密集网络作为自动编码器.确实找到正确的瓶颈取决于数据.但是，如果您将瓶颈设置为与输入后，您将获得身份功能.
是的，假设不涉及非线性，这会使事情变得更难(请参阅 我对该声明感到好奇:始终使用不同的时间步长而不是时间步长"它似乎通过将所有功能紧密结合在一起，但我不明白为什么这是关键吗?有一个更大的模型似乎是解决方案，减法只是有帮助. 这里的关键确实是增加模型的容量.减法实际上取决于数据.让我们想象一个极端的情况: 神经网络会做什么(这里最简单的是什么)?可能会将这个 如果减去，该怎么办?对于每个时间步，整个神经网络的损失都在 是的，在某种意义上，它与规范化有关. Autoencoder underfits timeseries reconstruction and just predicts average value. Here is a summary of my attempt at a sequence-to-sequence autoencoder. This image was taken from this paper: https://arxiv.org/pdf/1607.00148.pdf
Encoder: Standard LSTM layer. Input sequence is encoded in the final hidden state. Decoder: LSTM Cell (I think!). Reconstruct the sequence one element at a time, starting with the last element Decoder algorithm is as follows for a sequence of length
Here is my implementation, starting with the encoder: Decoder class: Bringing the two together: And here's my training function:
Large dataset of events scraped from the news (ICEWS). Various categories exist that describe each event. I initially one-hot encoded these variables, expanding the data to 274 dimensions. However, in order to debug the model, I've cut it down to a single sequence that is 14 timesteps long and only contains 5 variables. Here is the sequence I'm trying to overfit: And here is the custom
The model only learns the average, no matter how complex I make the model or now long I train it. Predicted/Reconstruction:
Actual: This problem is identical to the one discussed in this question: LSTM autoencoder always returns the average of the input sequence The problem in that case ended up being that the objective function was averaging the target timeseries before calculating loss. This was due to some broadcasting errors because the author didn't have the right sized inputs to the objective function. In my case, I do not see this being the issue. I have checked and double checked that all of my dimensions/sizes line up. I am at a loss. 100 个时间步，单个功能 10000 1 1 或更小的更改作为噪声丢弃，而只是预测所有这些 1000 (尤其是在某些正则化的情况下)，因为被 1/1000 不多. [0，1] 的余量内，而不是在 [0，1001] 的余量内，因此更容易出错.TLDR:
Question Set-up:
x[N].N:
hs[N]: Just use encoder final hidden state.x[N]= w.dot(hs[N]) + b.x[i]= w.dot(hs[i]) + bx[i] and hs[i] as inputs to LSTMCell to get x[i-1] and hs[i-1]Minimum Working Example:
class SeqEncoderLSTM(nn.Module):
def __init__(self, n_features, latent_size):
super(SeqEncoderLSTM, self).__init__()
self.lstm = nn.LSTM(
n_features,
latent_size,
batch_first=True)
def forward(self, x):
_, hs = self.lstm(x)
return hs
class SeqDecoderLSTM(nn.Module):
def __init__(self, emb_size, n_features):
super(SeqDecoderLSTM, self).__init__()
self.cell = nn.LSTMCell(n_features, emb_size)
self.dense = nn.Linear(emb_size, n_features)
def forward(self, hs_0, seq_len):
x = torch.tensor([])
# Final hidden and cell state from encoder
hs_i, cs_i = hs_0
# reconstruct first element with encoder output
x_i = self.dense(hs_i)
x = torch.cat([x, x_i])
# reconstruct remaining elements
for i in range(1, seq_len):
hs_i, cs_i = self.cell(x_i, (hs_i, cs_i))
x_i = self.dense(hs_i)
x = torch.cat([x, x_i])
return x
class LSTMEncoderDecoder(nn.Module):
def __init__(self, n_features, emb_size):
super(LSTMEncoderDecoder, self).__init__()
self.n_features = n_features
self.hidden_size = emb_size
self.encoder = SeqEncoderLSTM(n_features, emb_size)
self.decoder = SeqDecoderLSTM(emb_size, n_features)
def forward(self, x):
seq_len = x.shape[1]
hs = self.encoder(x)
hs = tuple([h.squeeze(0) for h in hs])
out = self.decoder(hs, seq_len)
return out.unsqueeze(0)
def train_encoder(model, epochs, trainload, testload=None, criterion=nn.MSELoss(), optimizer=optim.Adam, lr=1e-6, reverse=False):
device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f'Training model on {device}')
model = model.to(device)
opt = optimizer(model.parameters(), lr)
train_loss = []
valid_loss = []
for e in tqdm(range(epochs)):
running_tl = 0
running_vl = 0
for x in trainload:
x = x.to(device).float()
opt.zero_grad()
x_hat = model(x)
if reverse:
x = torch.flip(x, [1])
loss = criterion(x_hat, x)
loss.backward()
opt.step()
running_tl += loss.item()
if testload is not None:
model.eval()
with torch.no_grad():
for x in testload:
x = x.to(device).float()
loss = criterion(model(x), x)
running_vl += loss.item()
valid_loss.append(running_vl / len(testload))
model.train()
train_loss.append(running_tl / len(trainload))
return train_loss, valid_loss
Data:
tensor([[0.5122, 0.0360, 0.7027, 0.0721, 0.1892],
[0.5177, 0.0833, 0.6574, 0.1204, 0.1389],
[0.4643, 0.0364, 0.6242, 0.1576, 0.1818],
[0.4375, 0.0133, 0.5733, 0.1867, 0.2267],
[0.4838, 0.0625, 0.6042, 0.1771, 0.1562],
[0.4804, 0.0175, 0.6798, 0.1053, 0.1974],
[0.5030, 0.0445, 0.6712, 0.1438, 0.1404],
[0.4987, 0.0490, 0.6699, 0.1536, 0.1275],
[0.4898, 0.0388, 0.6704, 0.1330, 0.1579],
[0.4711, 0.0390, 0.5877, 0.1532, 0.2201],
[0.4627, 0.0484, 0.5269, 0.1882, 0.2366],
[0.5043, 0.0807, 0.6646, 0.1429, 0.1118],
[0.4852, 0.0606, 0.6364, 0.1515, 0.1515],
[0.5279, 0.0629, 0.6886, 0.1514, 0.0971]], dtype=torch.float64)
Dataset class:class TimeseriesDataSet(Dataset):
def __init__(self, data, window, n_features, overlap=0):
super().__init__()
if isinstance(data, (np.ndarray)):
data = torch.tensor(data)
elif isinstance(data, (pd.Series, pd.DataFrame)):
data = torch.tensor(data.copy().to_numpy())
else:
raise TypeError(f"Data should be ndarray, series or dataframe. Found {type(data)}.")
self.n_features = n_features
self.seqs = torch.split(data, window)
def __len__(self):
return len(self.seqs)
def __getitem__(self, idx):
try:
return self.seqs[idx].view(-1, self.n_features)
except TypeError:
raise TypeError("Dataset only accepts integer index/slices, not lists/arrays.")
Problem:
My research:
Other Things I've Tried
- I've tried this with varied sequence lengths from 7 timesteps to 100 time steps.
- I've tried with varied number of variables in the time series. I've tried with univariate all the way to all 274 variables that the data contains.
- I've tried with various
reductionparameters on thenn.MSELossmodule. The paper calls forsum, but I've tried bothsumandmean. No difference. - The paper calls for reconstructing the sequence in reverse order (see graphic above). I have tried this method using the
flipudon the original input (after training but before calculating loss). This makes no difference. - I tried making the model more complex by adding an extra LSTM layer in the encoder.
- I've tried playing with the latent space. I've tried from 50% of the input number of features to 150%.
- I've tried overfitting a single sequence (provided in the Data section above).
Question:
What is causing my model to predict the average and how do I fix it?
Okay, after some debugging I think I know the reasons.
TLDR
- You try to predict next timestep value instead of difference between current timestep and the previous one
- Your
hidden_featuresnumber is too small making the model unable to fit even a single sample
Analysis
Code used
Let's start with the code (model is the same):
import seaborn as sns
import matplotlib.pyplot as plt
def get_data(subtract: bool = False):
# (1, 14, 5)
input_tensor = torch.tensor(
[
[0.5122, 0.0360, 0.7027, 0.0721, 0.1892],
[0.5177, 0.0833, 0.6574, 0.1204, 0.1389],
[0.4643, 0.0364, 0.6242, 0.1576, 0.1818],
[0.4375, 0.0133, 0.5733, 0.1867, 0.2267],
[0.4838, 0.0625, 0.6042, 0.1771, 0.1562],
[0.4804, 0.0175, 0.6798, 0.1053, 0.1974],
[0.5030, 0.0445, 0.6712, 0.1438, 0.1404],
[0.4987, 0.0490, 0.6699, 0.1536, 0.1275],
[0.4898, 0.0388, 0.6704, 0.1330, 0.1579],
[0.4711, 0.0390, 0.5877, 0.1532, 0.2201],
[0.4627, 0.0484, 0.5269, 0.1882, 0.2366],
[0.5043, 0.0807, 0.6646, 0.1429, 0.1118],
[0.4852, 0.0606, 0.6364, 0.1515, 0.1515],
[0.5279, 0.0629, 0.6886, 0.1514, 0.0971],
]
).unsqueeze(0)
if subtract:
initial_values = input_tensor[:, 0, :]
input_tensor -= torch.roll(input_tensor, 1, 1)
input_tensor[:, 0, :] = initial_values
return input_tensor
if __name__ == "__main__":
torch.manual_seed(0)
HIDDEN_SIZE = 10
SUBTRACT = False
input_tensor = get_data(SUBTRACT)
model = LSTMEncoderDecoder(input_tensor.shape[-1], HIDDEN_SIZE)
optimizer = torch.optim.Adam(model.parameters())
criterion = torch.nn.MSELoss()
for i in range(1000):
outputs = model(input_tensor)
loss = criterion(outputs, input_tensor)
loss.backward()
optimizer.step()
optimizer.zero_grad()
print(f"{i}: {loss}")
if loss < 1e-4:
break
# Plotting
sns.lineplot(data=outputs.detach().numpy().squeeze())
sns.lineplot(data=input_tensor.detach().numpy().squeeze())
plt.show()
What it does:
get_dataeither works on the data your provided ifsubtract=Falseor (ifsubtract=True) it subtracts value of the previous timestep from the current timestep- Rest of the code optimizes the model until
1e-4loss reached (so we can compare how model's capacity and it's increase helps and what happens when we use the difference of timesteps instead of timesteps)
We will only vary HIDDEN_SIZE and SUBTRACT parameters!
NO SUBTRACT, SMALL MODEL
HIDDEN_SIZE=5SUBTRACT=False
In this case we get a straight line. Model is unable to fit and grasp the phenomena presented in the data (hence flat lines you mentioned).
1000 iterations limit reached
SUBTRACT, SMALL MODEL
HIDDEN_SIZE=5SUBTRACT=True
Targets are now far from flat lines, but model is unable to fit due to too small capacity.
1000 iterations limit reached
NO SUBTRACT, LARGER MODEL
HIDDEN_SIZE=100SUBTRACT=False
It got a lot better and our target was hit after 942 steps. No more flat lines, model capacity seems quite fine (for this single example!)
SUBTRACT, LARGER MODEL
HIDDEN_SIZE=100SUBTRACT=True
Although the graph does not look that pretty, we got to desired loss after only 215 iterations.
Finally
- Usually use difference of timesteps instead of timesteps (or some other transformation, see here for more info about that). In other cases, neural network will try to simply... copy output from the previous step (as that's the easiest thing to do). Some minima will be found this way and going out of it will require more capacity.
- When you use the difference between timesteps there is no way to "extrapolate" the trend from previous timestep; neural network has to learn how the function actually varies
- Use larger model (for the whole dataset you should try something like
300I think), but you can simply tune that one. - Don't use
flipud. Use bidirectional LSTMs, in this way you can get info from forward and backward pass of LSTM (not to confuse with backprop!). This also should boost your score
Questions
Okay, question 1: You are saying that for variable x in the time series, I should train the model to learn x[i] - x[i-1] rather than the value of x[i]? Am I correctly interpreting?
Yes, exactly. Difference removes the urge of the neural network to base it's predictions on the past timestep too much (by simply getting last value and maybe changing it a little)
Question 2: You said my calculations for zero bottleneck were incorrect. But, for example, let's say I'm using a simple dense network as an auto encoder. Getting the right bottleneck indeed depends on the data. But if you make the bottleneck the same size as the input, you get the identity function.
Yes, assuming that there is no non-linearity involved which makes the thing harder (see here for similar case). In case of LSTMs there are non-linearites, that's one point.
Another one is that we are accumulating timesteps into single encoder state. So essentially we would have to accumulate timesteps identities into a single hidden and cell states which is highly unlikely.
One last point, depending on the length of sequence, LSTMs are prone to forgetting some of the least relevant information (that's what they were designed to do, not only to remember everything), hence even more unlikely.
Is num_features * num_timesteps not a bottle neck of the same size as the input, and therefore shouldn't it facilitate the model learning the identity?
It is, but it assumes you have num_timesteps for each data point, which is rarely the case, might be here. About the identity and why it is hard to do with non-linearities for the network it was answered above.
One last point, about identity functions; if they were actually easy to learn, ResNets architectures would be unlikely to succeed. Network could converge to identity and make "small fixes" to the output without it, which is not the case.
I'm curious about the statement : "always use difference of timesteps instead of timesteps" It seem to have some normalizing effect by bringing all the features closer together but I don't understand why this is key ? Having a larger model seemed to be the solution and the substract is just helping.
Key here was, indeed, increasing model capacity. Subtraction trick depends on the data really. Let's imagine an extreme situation:
- We have
100timesteps, single feature - Initial timestep value is
10000 - Other timestep values vary by
1at most
What the neural network would do (what is the easiest here)? It would, probably, discard this 1 or smaller change as noise and just predict 1000 for all of them (especially if some regularization is in place), as being off by 1/1000 is not much.
What if we subtract? Whole neural network loss is in the [0, 1] margin for each timestep instead of [0, 1001], hence it is more severe to be wrong.
And yes, it is connected to normalization in some sense come to think about it.
这篇关于LSTM自动编码器问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
