如何在Google数据流管道中设置diskSourceImage [英] How to set diskSourceImage in google data flow pipeline

问题描述

我一直在尝试使用自定义图片来运行我的Google数据流管道。从 https://cloud.google.com/compute/docs/reference获取信息/ latest / images 我测试了以下代码片段：

I've been trying to use custom made images to run my google data flow pipeline. Given the information from https://cloud.google.com/compute/docs/reference/latest/images I've tested the following code snippets:

DataflowPipelineOptions options = PipelineOptionsFactory.create().as(DataflowPipelineOptions.class);
...
options.setDiskSourceImage("ubuntu-1504-vivid-v20150911");
options.setDiskSourceImage("projects/ubuntu-os-cloud/global/images/ubuntu-1504-vivid-v20150911");
options.setDiskSourceImage("https://www.googleapis.com/compute/beta/projects/ubuntu-os-cloud/global/images/ubuntu-1504-vivid-v20150911");

以上所有尝试都会导致以下错误：

all of the above tries led to the following error in my pipeline:

（b9c7b66a676906f4）：无法创建虚拟机。原因：（b9c7b66a67690aef）：错误：消息：字段'resource.disks [0] .initializeParams.sourceImage'：'[edited]'的值无效。必须是指向正确类型的Compute资源的URL HTTP代码：400

(b9c7b66a676906f4): Unable to create VMs. Causes: (b9c7b66a67690aef): Error: Message: Invalid value for field 'resource.disks[0].initializeParams.sourceImage': '[edited]'. Must be the URL to a Compute resource of the correct type HTTP Code: 400

推荐答案

使用具有Dataflow的自定义磁盘映像不是可行的选项。标志diskSourceImage已弃用，将在未来的SDK版本中删除。它不再被支持的原因是因为Dataflow服务依赖于VM映像中的版本化资源。因此，Dataflow需要控制VM映像，以便我们可以根据需要对其进行升级。如果用户提供自己的自定义图像，我们无法保持它们与Dataflow服务的要求保持同步。

Using a custom disk image with Dataflow is not a viable option. The flag diskSourceImage is deprecated and will be removed in a future SDK release. The reason it is no longer supported is because the Dataflow service relies on versioned resources in the VM image. So Dataflow needs control of the VM image so that we can upgrade it as necessary. If users supply their own custom images we have no way of keeping them in sync with the requirements of the Dataflow service.

如果您的自定义VM映像基于Dataflow映像那么您将能够使用该自定义映像执行作业，直到Dataflow VM映像的下一个版本发布。没有合理的方法可以让自定义图像与Dataflow的虚拟机映像保持同步，这样您就可以继续使用它。

If your custom VM image is based off a Dataflow image then you would be able to execute jobs using that custom image until the next release of a Dataflow VM image. There is no reasonable way in which you would be able to keep your custom images in sync with Dataflow's VM images so that you would be able to keep this working.

如果您想要定制虚拟机映像，请告诉我们为什么（例如，通过[e-mail]向我们发送电子邮件），以便我们可以提出替代解决方案，或者考虑将来支持您的使用案例。

If you would like to customize the VM image please let us know why (e.g. send us an email at [e-mail]) so we can either suggest an alternative solution or else consider supporting your use case in the future.

这篇关于如何在Google数据流管道中设置diskSourceImage的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！