NLP递归神经网络始终提供恒定值 [英] NLP Recurrent Neural Network always gives constant values

问题描述

我已根据观看的视频在TensorFlow中编写了一个简单的循环网络: https://youtu.be/vq2nnJ4g6N0?t=8546

I've written a simple recurrent network in TensorFlow based on this video that I watched: https://youtu.be/vq2nnJ4g6N0?t=8546

在视频中，RNN演示了通过使网络一次生成一个字符的单词来制作莎士比亚戏剧的过程.网络的输出在下一次迭代时反馈到输入中.

In the video the RNN is demonstrated to produce Shakespeare plays by having the network produce words one character at a time. The output of the network is fed back into the input on the next iteration.

这是我的网络图:

+--------------------------------+
|                                |
|    In:  H E L L O   W O R L <--+-----+
|         | | | | | | | | | |    |     |
|         V V V V V V V V V V    |     | Recursive feed
|         +-----------------+    |     |
+-> Hin ->|  RNN + Softmax  |-> Hout   |
          +-----------------+          |
          | | | | | | | | | |          |
     Out: V V V V V V V V V V          |
          E L L O   W O R L D ---------+
                            ^ Character predicted by the network

我希望网络至少能够正确地进行复制.不幸的是，我的网络始终为所有值(ASCII空格字符)输出32.我不确定是什么原因引起的...

I expect the network to at least do the copying bit correctly. Unfortunately my network always outputs 32 for all values (ASCII space character). I'm not sure what is causing the issue...

请帮助我使我的网络产生诗歌！

Please help me get my network producing poetry!

我的代码在这里: https://github.com/calebh/namepoet/blob/03f112ced94c3319055fbcc74a2d41 py

可以用Lorem Ipsum的几段内容代替语料库，以加快训练速度(网络具有相同的不良行为).

The corpus can be replaced by a few paragraphs of Lorem Ipsum to speed up training (the network has the same bad behavior).

推荐答案

我会尝试将代码运行更长的时间吗?您有batch_size = 10，sequence_size = 30和20个迭代，您的网络实际上总共可以看到6000个字符，也许学习率为0.001，这还远远不够初始化.

I'd try to run the code for longer time? You have batch_size = 10, sequence_size = 30 and 20 iterations, your network essentially has seen 6000 characters in total, maybe with a learning rate of 0.001, it wasn't enough to move away from your initialization.

因此，我尝试将学习率提高到非常高的值(例如1或100)，然后查看它是否开始输出不同的字母以确认您的实现是否正确.经过如此高学习率训练的网络通常根本不准确.

Hence, I'd try to raise the learning rate to a very high value (e.g. 1 or 100) and see if it starts outputting different letters to confirm your implementation is somewhat correct. A network trained with such a high learning rate is usually not going to be accurate at all.

这篇关于NLP递归神经网络始终提供恒定值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！