暂停Dataproc集群-Google Compute引擎 [英] Pausing Dataproc cluster - Google Compute engine

问题描述

是否有一种暂停Dataproc集群的方法，所以当我不主动运行spark-shell或spark-submit作业时，我不会被计费吗?此链接上的集群管理说明: https://cloud.google. com/sdk/gcloud/reference/beta/dataproc/clusters/

is there a way of pausing a Dataproc cluster so I don't get billed when I am not actively running spark-shell or spark-submit jobs ? The cluster management instructions at this link: https://cloud.google.com/sdk/gcloud/reference/beta/dataproc/clusters/

仅显示如何破坏群集，但是例如，我已经安装了spark cassandra连接器API.除了创建每次需要安装的映像之外，我唯一的选择是吗?

only show how to destroy a cluster but I have installed spark cassandra connector API for example. Is my only alternative to just creating an image that I'll need to install every time ?

推荐答案

通常，最好的做法是将用于自定义群集的步骤提取到一些安装脚本中，然后使用Dataproc的初始化操作，可轻松在集群部署期间自动执行安装.

In general, the best thing to do is to distill out the steps you used to customize your cluster into some setup scripts, and then use Dataproc's initialization actions to easily automate doing the installation during cluster deployment.

这样，如果您想要在多个并发的Dataproc集群上进行相同的设置，或者想要更改计算机类型或收到次要版本的错误，就可以轻松地重现自定义项，而无需人工参与修复了Dataproc偶尔发布的问题.

This way, you can easily reproduce the customizations without requiring manual involvement if you ever want, for example, to do the same setup on multiple concurrent Dataproc clusters, or want to change machine types, or receive sub-minor-version bug fixes that Dataproc releases occasionally.

目前，确实没有官方支持的暂停Dataproc集群的方法，很大程度上是因为能够进行可复制的集群部署以及下面列出的其他注意事项，这意味着99％的时间最好使用初始化-操作自定义，而不是就地暂停群集.也就是说，可能会有短期的黑客入侵，例如进入 Google Compute Engine页面，选择要暂停的Dataproc集群中的实例，然后单击停止"而不删除它们.

There's indeed no officially supported way of pausing a Dataproc cluster at the moment, in large part simply because being able to have reproducible cluster deployments along with several other considerations listed below means that 99% of the time it's better to use initialization-action customizations instead of pausing a cluster in-place. That said, there are possible short-term hacks, such as going into the Google Compute Engine page, selecting the instances that are part of the Dataproc cluster you want to pause, and clicking "stop" without deleting them.

仅当基础实例正在运行时才发生Compute Engine每小时收费和Dataproc的每vCPU收费，因此，即使您手动停止"了实例，您也不会产生Dataproc或Compute Engine的实例小时收费. Dataproc仍将集群列为"RUNNING"，尽管有警告，如果您进入Dataproc集群摘要页面的"VM Instances"选项卡，将会看到警告.

The Compute Engine hourly charges and Dataproc's per-vCPU charges are only incurred when the underlying instance is running, so while you've "stopped" the instances manually, you won't incur Dataproc or Compute Engine's instance-hour charges despite Dataproc still listing the cluster as "RUNNING", albeit with warnings that you'll see if you go to the "VM Instances" tab of the Dataproc cluster summary page.

您应该，然后只需从 Google Compute Engine页面页面以使集群再次运行，但是考虑以下警告很重要:

You should then be able to just click "start" from the Google Compute Engine page page to have the cluster running again, but it's important to consider the following caveats:

1. 集群有时可能无法再次启动到健康状态；所有使用本地SSD的操作都无法停止并重新启动，但是除此之外，如果关闭顺序不正确，甚至用户安装的设置可能已损坏，则Hadoop守护程序可能由于某种原因而失败，无法刷新磁盘上的重要内容.启动过程的方式未知.
2. 即使虚拟机停止"，它们也依赖于剩余的基础永久磁盘 ，因此即使在已暂停"的情况下，您仍将继续产生费用；如果我们假设每GB每月$ 0.04，并且每个Dataproc节点默认使用500GB磁盘，那么每个实例每小时将继续支付约$ 0.028/小时；通常，您只需将数据放入Google Cloud Storage中进行长期存储，而不是试图将其长期保存在Dataproc群集的HDFS上，便可以更轻松地访问数据，并且费用也更低.
3. 如果您过多地依赖于手动集群设置，那么在需要调整集群大小，更改机器类型或更改区域等情况下，重新执行操作将变得更加困难. ，通过Dataproc的初始化操作，您可以使用 Dataproc的集群扩展功能来自动调整集群大小为创建的新工作人员运行初始化操作.

1. The cluster may occasionally fail to start up into a healthy state again; anything using local SSDs already can't be stopped and started again cleanly, but beyond that, Hadoop daemons may have failed for whatever reason to flush something important to disk if the shutdown wasn't orderly, or even user-installed settings may have broken the startup process in unknown ways.
2. Even when VMs are "stopped", they depend on the underlying Persistent Disks remaining, so you'll continue to incur charges for those even while "paused"; if we assume $0.04 per GB-month, and a default 500GB disk per Dataproc node, that comes out to continuing to pay ~$0.028/hour per instance; generally your data will be more accessible and also cheaper to just put in Google Cloud Storage for long term storage rather than trying to keep it long-term on the Dataproc cluster's HDFS.
3. If you come to depend on a manual cluster setup too much, then it'll become much more difficult to re-do if you need to size up your cluster, or change machine types, or change zones, etc. In contrast, with Dataproc's initialization actions, you can use Dataproc's cluster scaling feature to resize your cluster and automatically run the initialization actions for new workers created.

这篇关于暂停Dataproc集群-Google Compute引擎的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！