在Google Cloud Machine Learning中读取档案 [英] reading files in google cloud machine learning
问题描述
我尝试使用gcloud ml-engine jobs submit training
在Google云ml-engine上运行 tensorflow-wavenet 但是当尝试读取json配置文件时,云作业崩溃了:
I tried to run tensorflow-wavenet on the google cloud ml-engine with gcloud ml-engine jobs submit training
but the cloud job crashed when it was trying to read the json configuration file:
with open(args.wavenet_params, 'r') as f:
wavenet_params = json.load(f)
arg.wavenet_params
只是我上载到google cloud storage bucket的json文件的文件路径.文件路径如下所示:gs://BUCKET_NAME/FILE_PATH.json
.
arg.wavenet_params
is simply a file path to a json file which I uploaded to the google cloud storage bucket. The file path looks like this: gs://BUCKET_NAME/FILE_PATH.json
.
我仔细检查了文件路径是否正确,并且由于我注释掉了其他所有内容,因此我确定这部分是导致崩溃的原因.
I double-checked that the file path is correct and I'm sure that this part is responsible for the crash since I commented out everything else.
崩溃日志文件没有提供有关已发生事件的太多信息:
The crash log file doesn't give much information about what has happened:
Module raised an exception for failing to call a subprocess Command '['python', '-m', u'gcwavenet.train', u'--data_dir', u'gs://wavenet-test-data/VCTK-Corpus-Small/', u'--logdir_root', u'gs://wavenet-test-data//gcwavenet10/logs']' returned non-zero exit status 1.
我用f.close()
替换了wavenet_params = json.load(f)
,但仍然得到相同的结果.
I replaced wavenet_params = json.load(f)
by f.close()
and I still get the same result.
当我使用gcloud ml-engine local train
在本地运行时,一切正常.
Everything works when I run it locally with gcloud ml-engine local train
.
我认为问题通常是使用gcloud ml-engine
读取文件,或者我无法使用gs://BUCKET_NAME/FILE_PATH
从python文件中访问Google云存储桶.
I think the problem is with reading files with gcloud ml-engine
in general or that I can't access the google cloud bucket from within a python file with gs://BUCKET_NAME/FILE_PATH
.
推荐答案
Python的open
函数无法从GCS读取文件.您将需要使用能够执行此操作的库. TensorFlow包括一个这样的库:
Python's open
function cannot read files from GCS. You will need to use a library capable of doing so. TensorFlow includes one such library:
import tensorflow as tf
from tensorflow.python.lib.io import file_io
with file_io.FileIO(args.wavenet_params, 'r') as f:
wavenet_params = json.load(f)
这篇关于在Google Cloud Machine Learning中读取档案的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!