TextIO.使用模式{}从GCS读取多个文件 [英] TextIO. Read multiple files from GCS using pattern {}
问题描述
我尝试使用以下内容
TextIO.Read.from("gs://xyz.abc/xxx_{2017-06-06,2017-06-06}.csv")
按照我的说法,这种模式不起作用
That pattern didn't work, as I get
java.lang.IllegalStateException: Unable to find any files matching StaticValueProvider{value=gs://xyz.abc/xxx_{2017-06-06,2017-06-06}.csv}
即使那两个文件确实存在.我尝试使用类似的表达式处理本地文件
Even though those 2 files do exist. And I tried with a local file using a similar expression
TextIO.Read.from("somefolder/xxx_{2017-06-06,2017-06-06}.csv")
那确实很好.
我本以为GCS中的文件会支持各种glob,但不会.这是为什么?有没有完成我想要的东西的地方?
I would've thought there would be support for all kinds of globs for files in GCS, but nope. Why is that? is there away to accomplish what I'm looking for?
推荐答案
除了Scott的建议和您对他的回答的评论之外,这可能是另一个选择:
This may be another option, in addition to Scott's suggestion and your comment on his answer:
您可以定义一个列表,其中包含要读取的路径,然后在其上进行迭代,以通常的方式创建许多PCollection:
You can define a list with the paths you want to read and then iterate over it, creating a number of PCollections in the usual way:
PCollection<String> events1 = p.apply(TextIO.Read.from(path1));
PCollection<String> events2 = p.apply(TextIO.Read.from(path2));
然后创建一个 PCollectionList :
PCollectionList<String> eventsList = PCollectionList.of(events1).and(events2);
然后将该列表展平到您的PCollection中作为主要输入:
And then flatten this list into your PCollection for your main input:
PCollection<String> events = eventsList.apply(Flatten.pCollections());
这篇关于TextIO.使用模式{}从GCS读取多个文件的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!