张量流中未完全连接的层 [英] Not fully connected layer in tensorflow

问题描述

我想创建一个网络，在该网络中，输入层中的节点仅连接到下一层中的某些节点.这是一个小例子:

I want to create a network where in the input layer nodes are just connected to some nodes in the next layer. Here is a small example:

到目前为止，我的解决方案是将i1和h1之间的边的权重设置为零，并且在每个优化步骤之后，我将权重乘以一个矩阵(我称此矩阵掩码矩阵)，其中每个项除在i1和h1之间的边的权重输入外，其为1. (请参见下面的代码)

My solution so far is that I set the weight of the edge between i1 and h1 to zero and after every optimization step I multiply the weights with a matrix (I call this matrix mask matrix) in which every entry is 1 except the entry of the weight of the edge between i1 and h1. (See code below)

这种方法正确吗?还是对GradientDescent有影响?是否有另一种方法可以在TensorFlow中创建这种网络?

Is this approach right? Or does this have a affect on the GradientDescent? Is there another approach to create this kind of a network in TensorFlow?

import tensorflow as tf
import tensorflow.contrib.eager as tfe
import numpy as np

tf.enable_eager_execution()


model = tf.keras.Sequential([
  tf.keras.layers.Dense(2, activation=tf.sigmoid, input_shape=(2,)),  # input shape required
  tf.keras.layers.Dense(2, activation=tf.sigmoid)
])


#set the weights
weights=[np.array([[0, 0.25],[0.2,0.3]]),np.array([0.35,0.35]),np.array([[0.4,0.5],[0.45, 0.55]]),np.array([0.6,0.6])]

model.set_weights(weights)

model.get_weights()

features = tf.convert_to_tensor([[0.05,0.10 ]])
labels =  tf.convert_to_tensor([[0.01,0.99 ]])


mask =np.array([[0, 1],[1,1]])

#define the loss function
def loss(model, x, y):
  y_ = model(x)
  return tf.losses.mean_squared_error(labels=y, predictions=y_)

#define the gradient calculation
def grad(model, inputs, targets):
  with tf.GradientTape() as tape:
    loss_value = loss(model, inputs, targets)
  return loss_value, tape.gradient(loss_value, model.trainable_variables) 

#create optimizer an global Step
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01)
global_step = tf.train.get_or_create_global_step()


#optimization step
loss_value, grads = grad(model, features, labels)
optimizer.apply_gradients(zip(grads, model.variables),global_step)

#masking the optimized weights 
weights=(model.get_weights())[0]
masked_weights=tf.multiply(weights,mask)
model.set_weights([masked_weights])

推荐答案

如果您正在寻找提供的特定示例的解决方案，则可以简单地使用tf.keras Functional API并定义两个密集层，其中一层与之连接上一层中的两个神经元，另一层仅连接到其中一个神经元:

If you are looking for a solution for the specific example you provided, you can simply use tf.keras Functional API and define two Dense layers where one is connected to both neurons in the previous layer and the other one is only connected to one of the neurons:

from tensorflow.keras.layer import Input, Lambda, Dense, concatenate
from tensorflow.keras.models import Model

inp = Input(shape=(2,))
inp2 = Lambda(lambda x: x[:,1:2])(inp)   # get the second neuron 

h1_out = Dense(1, activation='sigmoid')(inp2)  # only connected to the second neuron
h2_out = Dense(1, activation='sigmoid')(inp)  # connected to both neurons
h_out = concatenate([h1_out, h2_out])

out = Dense(2, activation='sigmoid')(h_out)

model = Model(inp, out)

# simply train it using `fit`
model.fit(...)

这篇关于张量流中未完全连接的层的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！