Java Apache Beam - 保存文件“本地"通过使用 DataflowRunner [英] Java Apache Beam - save file "LOCALY" by using DataflowRunner

问题描述

Can send the java code but currently, it's not necessary.

我有一个问题，当我以(DirectRunner - 使用 Google VM 实例)运行作业时，它工作正常，因为它将信息保存到本地文件并继续......

I have an issue as when I run the job as (DirectRunner - using Google VM Instance) it is working fine, as it saves information to the local file and carries on...

尝试使用 (DataflowRunner) 时出现问题，以及我收到的错误:

The problem appears when trying to use (DataflowRunner), and the error which I receive:

java.nio.file.NoSuchFileExtension: XXXX.csv
.....
.....
XXXX.csv could not be delete.

它可以被删除，因为它甚至没有创建.

It could be deleted as it not even created.

问题-通过DataflowRunner运行时如何在本地写入文件??

Problem - how to write the file locally when running through DataflowRunner??

P.S. 使用 Apache Beam

P.S. Using Apache Beam

管道(部分代码)- 从 BigQuery 读取数据并将数据存储到 Google 存储(特殊字符问题)

推荐答案

AFAIK 当它作为数据流实例运行时，您必须将文件写入 GCS 服务(又名存储桶)而不是本地磁盘.

AFAIK when it is ran as a dataflow instance, you have to write file to GCS service (aka storage bucket) rather than local disk.

你已经尝试过了吗?创建存储桶:https://cloud.google.com/storage/docs/creating-桶

Did you try that already? to create storage bucket: https://cloud.google.com/storage/docs/creating-buckets

这篇关于Java Apache Beam - 保存文件“本地"通过使用 DataflowRunner的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！