AdamOptimizer 的问题 [英] Issue with AdamOptimizer

问题描述

我正在使用一个简单的网络，我正在尝试使用 AdamOptimizer 来最小化 Q 学习环境中的损失.

I'm using a simple network and i'm trying to use AdamOptimizer to minimize the loss in a Q learning contexte.

代码如下:

### DATASET IMPORT
from DataSet import *

### NETWORK
state_size      = STATE_SIZE
stack_size      = STACK_SIZE
action_size     = ACTION_SIZE
learning_rate   = LEARNING_RATE
hidden_tensors  = HIDDEN_TENSORS
gamma           = GAMMA

import tensorflow as tf
import numpy as np

class NNetwork:       
    def __init__(self, name='NNetwork'):

        # Initialisations
        self.state_size     = state_size
        self.action_size    = action_size
        self.model          = tf.keras.models.Sequential()
        self.optimizer      = tf.keras.optimizers.Adam(learning_rate)

        # Network shaping
        self.model.add(tf.keras.layers.Dense(self.state_size,   activation='relu',      kernel_initializer='glorot_uniform'))
        self.model.add(tf.keras.layers.Dense(hidden_tensors,    activation='relu',      kernel_initializer='glorot_uniform'))
        self.model.add(tf.keras.layers.Dense(action_size,       activation='linear',    kernel_initializer='glorot_uniform'))

    # Prediction function (return Q_values)
    def get_outputs(self, inputs):
        inputs = tf.convert_to_tensor(inputs, dtype=tf.float32)
        return self.model.predict(inputs)

    # Optimization of the network
    def optimize(self, state, action, reward, next_state):
        next_Q_values   = self.get_outputs(next_state)
        target_Q        = reward + gamma * np.max(next_Q_values)
        curent_Q        = tf.reduce_sum(tf.multiply(self.get_outputs(state), action))
        loss           = tf.square(target_Q - curent_Q)
        self.optimizer.minimize(tf.convert_to_tensor(loss), self.model.trainable_variables)



B = NNetwork('b')
print(B.get_outputs([[0.12, 0.59]]))

B.optimize([[0.12, 0.59]], [1, 0, 0, 0, 0, 0, 0], 100000000, [[0.13, 0.58]])
print(B.get_outputs([[0.12, 0.59]]))

所以我的问题是:

当我执行这段代码时，我得到了这个:

When i execute this code i got this :

[[-0.00105272 0.02356465 -0.01908724 -0.03868931 0.01585
0.024270340.00203115]] 回溯(最近一次调用最后一次):文件.\DQNet.py"，第 69 行，在B.optimize([[0.12, 0.59]], [1, 0, 0, 0, 0, 0, 0], 100000000, [[0.13, 0.58]]) 文件.\DQNet.py"，第62行，在优化tf.keras.optimizers.Adam(learning_rate).minimize(tf.convert_to_tensor(10),self.model.trainable_variables) 文件C:\Users\Odeven poste1\Documents[Python-3.6.8\python-3.6.8.amd64\lib\site-packages\tensorflow\python\keras\optimizer_v2\optimizer_v2.py",第 296 行，最小化loss, var_list=var_list, grad_loss=grad_loss) 文件C:\Users\Odeven poste1\Documents[Python-3.6.8\python-3.6.8.amd64\lib\site-packages\tensorflow\python\keras\optimizer_v2\optimizer_v2.py",第 328 行，在 _compute_gradients 中loss_value = loss() TypeError: 'tensorflow.python.framework.ops.EagerTensor' 对象不可调用

[[-0.00105272 0.02356465 -0.01908724 -0.03868931 0.01585
0.02427034 0.00203115]] Traceback (most recent call last): File ".\DQNet.py", line 69, in B.optimize([[0.12, 0.59]], [1, 0, 0, 0, 0, 0, 0], 100000000, [[0.13, 0.58]]) File ".\DQNet.py", line 62, in optimize tf.keras.optimizers.Adam(learning_rate).minimize(tf.convert_to_tensor(10), self.model.trainable_variables) File "C:\Users\Odeven poste 1\Documents[Python-3.6.8\python-3.6.8.amd64\lib\site-packages\tensorflow\python\keras\optimizer_v2\optimizer_v2.py", line 296, in minimize loss, var_list=var_list, grad_loss=grad_loss) File "C:\Users\Odeven poste 1\Documents[Python-3.6.8\python-3.6.8.amd64\lib\site-packages\tensorflow\python\keras\optimizer_v2\optimizer_v2.py", line 328, in _compute_gradients loss_value = loss() TypeError: 'tensorflow.python.framework.ops.EagerTensor' object is not callable

这意味着我的网络可以正常运行，因为我得到了 Q 值，但是当我尝试调用我的优化"函数时，我在线上出现错误:

That mean that my network is functional because i got Q values but when i try to call my 'optimize' function i got an error on the line :

self.optimizer.minimize(tf.convert_to_tensor(loss), self.model.trainable_variables)

我真的不明白为什么我会收到这个错误:

And i really don't understand why i got this error :

'tensorflow.python.framework.ops.EagerTensor' 对象不可调用

'tensorflow.python.framework.ops.EagerTensor' object is not callable

因为我很确定我必须给最小化函数的损失"参数应该是张量...

Because i'm pretty sure that the 'loss' parameter i have to give to the minimize function should be a Tensor...

推荐答案

在 TF2 中，最小化方法的 loss 参数必须是 Python 可调用的.

In TF2 the loss parameter of the minimize method must be a Python callable.

因此，您可以将损失定义更改为:

Thus, you can change your loss definition to:

def loss():
    return tf.square(target_Q - curent_Q)

并在不将其转换为张量的情况下使用它:

and use it without converting it to a Tensor:

self.optimizer.minimize(loss, self.model.trainable_variables)

这篇关于AdamOptimizer 的问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！