我在Datalab中使用Python3，无法将表示Google Cloud Storage存储桶中文件的字符串列表作为带有tensorflow的feed_dict进行输入 [英] Using Python3 in Datalab, I can't feed a list of strings which represent files in a Google Cloud Storage bucket as a feed_dict with tensorflow

问题描述

我是tf的新手，我在尝试处理某些文件时遇到问题.这是一段代码摘录.

I am new to tf and I have a problem where I am trying to process some files. Here is a excerpt of code.

xlabel_to_files_list_map['dog_bark'] # subset of data with two files

# result
['gs://some_bucket/some_dir/data/dog_bark/100652.mp3', 'gs://some_bucket/some_dir/dog_bark/100795.mp3']

在这里，我只是试图通过简单的图形来处理这些字符串:

Here is where I am simply trying to get these strings to be processed by a simple graph:

file_to_process = tf.placeholder(tf.string)

audio_binary_remote = tf.gfile.Open(file_to_process, 'rb').read()

waveform = tf.contrib.ffmpeg.decode_audio(audio_binary_remote, file_format='mp3', samples_per_second=44100, channel_count=2)


with tf.Session() as sess:
 result = sess.run(waveform, feed_dict={
 file_to_process: xlabel_to_files_list_map['dog_bark']
 })
#print (result)

这导致

TypeError: Expected binary or unicode string, got <tf.Tensor 'Placeholder_9:0' shape=<unknown> dtype=string>

FWIW，此方法有效

FWIW, this works

a_string = tf.placeholder(tf.string) 
z = a_string 
with tf.Session() as sess: 
    result = sess.run(z, feed_dict={a_string: ['one', 'two', 'three']}) 
print(result)

这导致

['one' 'two' 'three']

一个简单的例子是一个字符串列表.使用哈希映射值部分(它是字符串列表)的更复杂的示例.我不确定为什么它不能与第二个示例相似.

The simple example which works is of a list of strings. The more complex example which uses a hash map value part which is a list of strings. I'm not sure why it doesn't work similar to the second example.

另一种方法

我试图用另一种方式做到这一点.这次，我尝试构建结果列表，然后处理该列表.这也失败了.它没有产生错误.它只是给出了空白的结果.

I tried to do this another way. This time I tried to build a list of results and then process the list. This also failed. It did not produce an error. It simply gave blank results.

waveform_tensor_list = []
for a_file in dir_to_selected_files_list_map['gs://some_bucket/some_dir/dog_bark/']:
  #print (a_file)
  waveform = tf.contrib.ffmpeg.decode_audio(a_file, file_format='mp3', samples_per_second=44100, channel_count=2)
  waveform_tensor_list.append(waveform)

此单元格的输出立即看起来是错误的，但格式正确:

The output from this cell looks wrong immediately but in the proper form:

waveform_tensor_list

导致:

[<tf.Tensor 'DecodeAudioV2_7:0' shape=(?, 2) dtype=float32>,
 <tf.Tensor 'DecodeAudioV2_8:0' shape=(?, 2) dtype=float32>,
 stuff deleted,
 <tf.Tensor 'DecodeAudioV2_22:0' shape=(?, 2) dtype=float32>,
 <tf.Tensor 'DecodeAudioV2_23:0' shape=(?, 2) dtype=float32>]

然后评估图形，即:

with tf.Session() as sess:
  result = sess.run(waveform_tensor_list)
  print (result)

此单元格的输出为:

[array([], shape=(0, 0), dtype=float32), array([], shape=(0, 0), dtype=float32), array([], shape=(0, 0), dtype=float32), array([], shape=(0, 0), dtype=float32), array([], shape=(0, 0), dtype=float32), array([], shape=(0, 0), dtype=float32), array([], shape=(0, 0), dtype=float32), array([], shape=(0, 0), dtype=float32), array([], shape=(0, 0), dtype=float32), array([], shape=(0, 0), dtype=float32), array([], shape=(0, 0), dtype=float32), array([], shape=(0, 0), dtype=float32), array([], shape=(0, 0), dtype=float32), array([], shape=(0, 0), dtype=float32), array([], shape=(0, 0), dtype=float32), array([], shape=(0, 0), dtype=float32), array([], shape=(0, 0), dtype=float32)]

推荐答案

tf.gfile.Open 不是TensorFlow操作.换句话说，它不会向图形添加操作以打开文件.

tf.gfile.Open isn't a TensorFlow operation. In other words, it does not add operations to the graph to open the file.

tf.gfile.Open 是类

tf.gfile.Open is an alias for the class tf.gfile.GFile. So the line tf.gfile.Open(<foo>) is invoking tf.gfile.GFile.__init__ which expects the first argument to be a Python string, not tf.Tensor of strings (which is what tf.placeholder(tf.string) returns).

您在这里有几个选择:

raw_data = tf.placeholder(tf.string)
waveform = tf.contrib.ffmpeg.decode_audio(raw_data, file_format='mp3', samples_per_second=44100, channel_count=2)

with tf.Session() as sess:
    for file in xlabel_to_files_list_map['dog_bark']:
      result = sess.run(waveform, feed_dict={raw_data: tf.gfile.GFile(file, 'rb').read()})

打开并读取图中的文件

(使用 tf.data 类来设置输入处理" )

Open and read the file in the graph

(Using the tf.data classes to setup "input processing")

filenames = xlabel_to_files_list_map['dog_bark']
dataset = tf.data.Dataset.from_tensor_slices(filenames).map(lambda x: tf.read_file(x))

raw_data = dataset.make_one_shot_iterator().get_next()
waveform =  tf.contrib.ffmpeg.decode_audio(raw_data, file_format='mp3', samples_per_second=44100, channel_count=2)

with tf.Session() as sess:
    for _ in filenames:
        result = sess.run(waveform)

使用急切执行

(请参见TensorFlow入门指南的研究和实验"部分研究和实验部分)

这可能有助于减轻图形中的内容与Python中发生的事情之间的某些混淆.

Which may help reduce some of the confusion between what's in the graph and what's happening in Python.

tf.enable_eager_execution()
for filename in xlabel_to_files_list_map['dog_bark']:
    result = tf.contrib.ffmpeg.decode_audio(tf.gfile.GFile(filename, 'rb').read(), file_format='mp3', samples_per_second=44100, channel_count=2)

希望有帮助！

这篇关于我在Datalab中使用Python3，无法将表示Google Cloud Storage存储桶中文件的字符串列表作为带有tensorflow的feed_dict进行输入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！