张量流中的非全连接层 [英] Not fully connected layer in tensorflow

问题描述

我想创建一个网络，其中输入层中的节点仅连接到下一层中的某些节点.这是一个小例子:

I want to create a network where in the input layer nodes are just connected to some nodes in the next layer. Here is a small example:

到目前为止，我的解决方案是将 i1 和 h1 之间的边的权重设置为零，并且在每个优化步骤之后，我将权重与矩阵相乘(I称这个矩阵掩码矩阵)，其中除了i1和h1之间边的权重的条目外，每个条目都是1.(见下方代码)

My solution so far is that I set the weight of the edge between i1 and h1 to zero and after every optimization step I multiply the weights with a matrix (I call this matrix mask matrix) in which every entry is 1 except the entry of the weight of the edge between i1 and h1. (See code below)

这种方法对吗?或者这对 GradientDescent 有影响吗?是否有另一种方法可以在 TensorFlow 中创建这种网络?

Is this approach right? Or does this have a affect on the GradientDescent? Is there another approach to create this kind of a network in TensorFlow?

import tensorflow as tf
import tensorflow.contrib.eager as tfe
import numpy as np

tf.enable_eager_execution()


model = tf.keras.Sequential([
  tf.keras.layers.Dense(2, activation=tf.sigmoid, input_shape=(2,)),  # input shape required
  tf.keras.layers.Dense(2, activation=tf.sigmoid)
])


#set the weights
weights=[np.array([[0, 0.25],[0.2,0.3]]),np.array([0.35,0.35]),np.array([[0.4,0.5],[0.45, 0.55]]),np.array([0.6,0.6])]

model.set_weights(weights)

model.get_weights()

features = tf.convert_to_tensor([[0.05,0.10 ]])
labels =  tf.convert_to_tensor([[0.01,0.99 ]])


mask =np.array([[0, 1],[1,1]])

#define the loss function
def loss(model, x, y):
  y_ = model(x)
  return tf.losses.mean_squared_error(labels=y, predictions=y_)

#define the gradient calculation
def grad(model, inputs, targets):
  with tf.GradientTape() as tape:
    loss_value = loss(model, inputs, targets)
  return loss_value, tape.gradient(loss_value, model.trainable_variables) 

#create optimizer an global Step
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01)
global_step = tf.train.get_or_create_global_step()


#optimization step
loss_value, grads = grad(model, features, labels)
optimizer.apply_gradients(zip(grads, model.variables),global_step)

#masking the optimized weights 
weights=(model.get_weights())[0]
masked_weights=tf.multiply(weights,mask)
model.set_weights([masked_weights])

推荐答案

如果您正在为您提供的特定示例寻找解决方案，您可以简单地使用 tf.keras 函数式 API 并定义两个密集层，其中一个连接到前一层的两个神经元，另一个只连接到一个神经元:

If you are looking for a solution for the specific example you provided, you can simply use tf.keras Functional API and define two Dense layers where one is connected to both neurons in the previous layer and the other one is only connected to one of the neurons:

from tensorflow.keras.layer import Input, Lambda, Dense, concatenate
from tensorflow.keras.models import Model

inp = Input(shape=(2,))
inp2 = Lambda(lambda x: x[:,1:2])(inp)   # get the second neuron 

h1_out = Dense(1, activation='sigmoid')(inp2)  # only connected to the second neuron
h2_out = Dense(1, activation='sigmoid')(inp)  # connected to both neurons
h_out = concatenate([h1_out, h2_out])

out = Dense(2, activation='sigmoid')(h_out)

model = Model(inp, out)

# simply train it using `fit`
model.fit(...)

这篇关于张量流中的非全连接层的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！