神经网络并行训练，最好使用Hadoop或gpu？ [英] Neural Network training in parallel, better to use Hadoop or a gpu?
问题描述
Thanks
我很幸运地在一个涉及这两种培训方法的实验室工作网络，虽然两者在非常计算昂贵的设置中都很有用，但计算瓶颈的位置通常决定使用哪种方法。
使用分布式系统（例如HADOOP）
当网络很大时，这非常有用，以至于在训练中涉及的矩阵乘法变得笨拙传统的个人电脑。当您有严格的时间限制时（例如在线培训），此问题尤其普遍，否则HADOOP实施的麻烦不值得（仅在夜间运行网络）。如果您考虑的是HADOOP，因为您想要摆弄网络参数，而不必等待一天再摆弄一些东西（我的实验室经常出现这种情况），那么只需在不同的机器上运行具有不同参数的网络的多个实例即可。这样你就可以在不处理实际分布式计算的情况下使用你的集群。
示例： 重新培训一个网络来查找图像中的人数。您决定让该程序从Google中提取随机图片，而不是预先定义的一组训练示例（图片 - 人数对）。网络正在处理图像时，您必须查看图像并提供关于图像中实际有多少人的反馈。由于这是图像处理，您的网络规模可能在数百万个单位的规模上。而且，由于您实时提供反馈，网络计算的速度很重要。因此，您应该投资于分布式实现。
在GPU上训练网络
如果主要计算瓶颈不是网络大小，而是训练集的大小（尽管网络通常仍然很大），这是正确的选择。由于GPU对于在大量数据集中应用相同的矢量/矩阵操作的情况非常理想，因此它们主要用于可以使用非常大批量的批量训练例如：
您正在训练一个网络来回答以自然语言提出的问题。你有一个问题 - 答案对的巨大数据库，并不介意每10000个问题只更新其权重的网络。有了这么大的批量大小，可能也是一个相当大的网络，基于GPU的实现将是一个好主意。
I need to train a neural network with 2-4 hidden layers, not sure yet on the structure of the actual net. I was thinking to train it using Hadoop map reduce (cluster of 12 pcs) or a gpu in order to get faster results. What do you think it would be better ? Also are there any available libraries that have these already implemented? Thanks
I've been luckily to work in a lab which has dabbled in both of these methods for training networks, and while both are useful in very computationally expensive settings, the location of the computational bottleneck usually determines which method to use.
Training a network using a distributed system (e.g. HADOOP)
This is useful when your network is large enough that the matrix multiplications involved in training become unwieldy on a traditional PC. This problem is particularly prevalent when you have harsh time constraints (e.g. online training), as otherwise the hassle of a HADOOP implementation isn't worth it (just run the network overnight). If you're thinking about HADOOP because you want to fiddle with network parameters and not have to wait a day before fiddling some more (frequently the case in my lab), then simply run multiple instances of the network with different parameters on different machines. That way you can make use of your cluster without dealing with actual distributed computation.
Example:
You're training a network to find the number of people in images. Instead of a predefined set of training examples (image-number of people pairs) you decide to have the program pull random images from Google. While the network is processing the image, you must view the image and provide feedback on how many people are actually in the image. Since this is image processing, your network size is probably on the scale of millions of units. And since you're providing the feedback in real time the speed of the network's computations matters. Thus, you should probably invest in a distributed implementation.
Training a network on a GPU
This is the right choice if the major computational bottleneck isn't the network size, but the size of the training set (though the networks are still generally quite large). Since GPUs are ideal for situations involving applying the same vector/matrix operation across a large number of data sets, they are mainly used when you can use batch training with a very large batch size.
Example:
You're training a network to answer questions posed in natural language. You have a huge database of question-answer pairs and don't mind the network only updating its weights every 10000 questions. With such a large batch size and presumably a rather large network as well, a GPU based implementation would be a good idea.
这篇关于神经网络并行训练，最好使用Hadoop或gpu？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
