如何比较不同Keras模型的权重? [英] How can I compare weights of different Keras models?

问题描述

我以.h5格式保存了许多模型.我想比较一下它们的特性，例如重量. 我没有任何想法，特别是如何以表格和数字的形式对它们进行适当的比较. 提前致谢.

I've saved numbers of models in .h5 format. I want to compare their characteristics such as weight. I don't have any Idea how I can appropriately compare them specially in the form of tables and figures. Thanks in advance.

推荐答案

体重自省是一项相当先进的工作，需要特定于模型的处理.权重的可视化在很大程度上是一项技术挑战，但是您对这些信息的处理又是另一回事-我将主要讨论前者，但涉及后者.

Weight-introspection is a fairly advanced endeavor, and requires model-specific treatment. Visualizing weights is a largely technical challenge, but what you do with that information's a different matter - I'll address largely the former, but touch upon the latter.

更新:我还建议有关权重，请参见RNN ，渐变和激活可视化.

Update: I also recommend See RNN for weights, gradients, and activations visualization.

可视化权重:一种方法如下:

1. 检索感兴趣层的权重. Ex :model.layers[1].get_weights()
2. 了解重量作用和尺寸. Ex :LSTM具有三组权重:kernel，recurrent和bias，每组都有不同的用途. 内部中的每个权重矩阵都是 gate 权重-输入，单元格，遗忘，输出.对于Conv图层，过滤器(dim0)，内核和步幅之间是有区别的.
3. 按(2)以有意义的方式组织权重矩阵以进行可视化. Ex :对于Conv而言，与LSTM不同，它并不是真的必须进行特定功能的处理，我们只需将内核权重和偏权重平化并在直方图中将其可视化即可.
4. 选择可视化方法:直方图，热图，散点图等-对于扁平化数据，直方图是最好的选择

1. Retrieve weights of layer of interest. Ex: model.layers[1].get_weights()
2. Understand weight roles and dimensionality. Ex: LSTMs have three sets of weights: kernel, recurrent, and bias, each serving a different purpose. Within each weight matrix are gate weights - Input, Cell, Forget, Output. For Conv layers, the distinction's between filters (dim0), kernels, and strides.
3. Organize weight matrices for visualization in a meaningful manner per (2). Ex: for Conv, unlike for LSTM, feature-specific treatment isn't really necessary, and we can simply flatten kernel weights and bias weights and visualize them in a histogram
4. Select visualization method: histogram, heatmap, scatterplot, etc - for flattened data, a histogram is the best bet

---

解释权重:几种方法是:

---

Interpreting weights: a few approaches are:

• 稀疏性:如果权重标准(平均")低，则模型稀疏.可能有好处，也可能没有好处.
• 健康:如果太多权重为零或接近零，则表明死神经元过多.这对于调试很有用，因为一旦图层处于这种状态，通常不会还原-因此应重新开始训练
• 稳定性:如果权重变化迅速而又大，或者如果有很多高价值的权重，则可能表示梯度性能受损，例如可以通过梯度裁剪或权重约束

• Sparsity: if weight norm ("average") is low, the model is sparse. May or may not be beneficial.
• Health: if too many weights are zero or near-zero, it's a sign of too many dead neurons; this can be useful for debugging, as once a layer's in such a state, it usually does not revert - so training should be restarted
• Stability: if weights are changing greatly and quickly, or if there are many high-valued weights, it may indicate impaired gradient performance, remedied by e.g. gradient clipping or weight constraints

模型比较:没有一种方法可以简单地并排查看来自不同模型的两个权重，并确定这是更好的".例如，如上所述，分别分析每个模型，然后决定哪个模型胜于其他模型.

Model comparison: there isn't a way for simply looking at two weights from separate models side-by-side and deciding "this is the better one"; analyze each model separately, for example as above, then decide which one's ups outweigh downs.

最终的决胜局将是验证性能-而且它也是更实际的一种.内容如下:

The ultimate tiebreaker, however, will be validation performance - and it's also the more practical one. It goes as:

1. 几种超参数配置的训练模型
2. 选择验证性能最佳的一个
3. 微调该模型(例如，通过其他超参数配置)

重量可视化应该主要作为调试或日志记录工具保存-简单地说，即使我们对神经网络有最新的了解，也不能仅通过查看重量就可以判断模型的概括性.

Weight visualization should be mainly kept as a debugging or logging tool - as, put simply, even with our best current understanding of neural networks one cannot tell how well the model will generalize just by looking at the weights.

建议:还可视化图层输出-请参见

Suggestion: also visualize layer outputs - see this answer and sample output at bottom.

视觉示例:

from tensorflow.keras.layers import Input, Conv2D, Dense, Flatten
from tensorflow.keras.models import Model

ipt = Input(shape=(16, 16, 16))
x   = Conv2D(12, 8, 1)(ipt)
x   = Flatten()(x)
out = Dense(16)(x)

model = Model(ipt, out)
model.compile('adam', 'mse')

X = np.random.randn(10, 16, 16, 16)  # toy data
Y = np.random.randn(10, 16)  # toy labels
for _ in range(10):
    model.train_on_batch(X, Y)

def get_weights_print_stats(layer):
    W = layer.get_weights()
    print(len(W))
    for w in W:
        print(w.shape)
    return W

def hist_weights(weights, bins=500):
    for weight in weights:
        plt.hist(np.ndarray.flatten(weight), bins=bins)

W = get_weights_print_stats(model.layers[1])
# 2
# (8, 8, 16, 12)
# (12,)

hist_weights(W)

---

---

Conv1D输出可视化 :(

这篇关于如何比较不同Keras模型的权重?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！