Python Dataflow SDK中的自定义机器类型 [英] Custom Machine Types in Python Dataflow SDK
问题描述
根据>是否可以使用一个用于Dataflow实例的自定义计算机?您可以通过将名称指定为custom-<number of cpus>-<memory in mb>
According to Is it possible to use a Custom machine for Dataflow instances? you can set the custom machine type for a dataflow operation by specifying the name as custom-<number of cpus>-<memory in mb>
但是答案是针对Java Api和旧的Dataflow版本,而不是新的Apache Beam实现和Python.
But that answer is for the Java Api and the old Dataflow version, not the new Apache Beam implementation and Python.
如果我在2.0.0 google-cloud-dataflow python API中提供了--worker_machine_type custom-8-5376
,则会出现以下错误:
If I supply --worker_machine_type custom-8-5376
in the 2.0.0 google-cloud-dataflow python API, I get the following error:
(4092fe7df5a10577):无法创建工作流程.请在几分钟后重试.如果仍然无法创建作业,请与客户支持联系.原因:(4092fe7df5a10596):无法获取以下机器的类型信息us-central1-f区域中的计算机类型custom-8-5376.请检查计算机类型和区域是否正确."
"(4092fe7df5a10577): The workflow could not be created. Please try again in a few minutes. If you are still unable to create a job please contact customer support. Causes: (4092fe7df5a10596): Unable to get machine type information for machine type custom-8-5376 in zone us-central1-f. Please check that machine type and zone are correct."
我还尝试在计算引擎中定义一个新的实例模板,并在--worker_machine_type
参数中提供该模板的名称,但这也不起作用.
I also tried defining a new instance template in the compute engine and supplying the name of that template in the --worker_machine_type
parameter, but that doesn't work, either.
如何使用自定义计算机类型在Dataflow 2.0.0上运行工作流?
How can you run a workflow on Dataflow 2.0.0 with a custom machine type?
推荐答案
每个自定义计算机类型doc: https://cloud.google.com/compute/docs/instances/creating-instance-with-custom-machine-type
per custom machine type doc: https://cloud.google.com/compute/docs/instances/creating-instance-with-custom-machine-type
每个vCPU的自定义计算机类型的每个vCPU的内存必须在0.9 GB和6.5 GB之间(包括两端).
The memory per vCPU of a custom machine type must be between 0.9 GB and 6.5 GB per vCPU, inclusive.
因此对于8个vCPU,最小为7424MiB.
So for 8 vCPUs, 7424MiB is the minimum.
能否请您再试一次?
这篇关于Python Dataflow SDK中的自定义机器类型的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!