Google Compute Engine根据队列长度自动缩放 [英] Google Compute Engine auto scaling based on queue length

问题描述

我们在Google Compute Engine上托管我们的基础架构，并正在研究Autoscaling的实例组。我们对队列中的二进制数据进行了大量批处理。在我们的例子中，这意味着：

1. 当一个worker正在处理数据时，CPU总是100％


2. 当队列为空时，我们要终止所有工作人员


3. 根据队列的长度，我们需要一定数量的工作人员

然而，我发现很难找到一种在Google Compute Engine上自动缩放此功能的方法，因为它们似乎可以扩展实例级别的指标，如CPU。来自文档：

并非所有自定义指标都可以被autoscaler使用。要选择一个
的有效自定义指标，该指标必须具有以下所有
属性：

• 成为每个实例的指标。


• 该度量标准必须是有效的利用率度量标准，这意味着度量标准中的数据可用于按比例放大或缩小虚拟机的数量。

如果我正确阅读文档，就很难在全局队列长度？

备份解决方案

1. 使用Google Cloud API使用实例API创建或销毁新工作人员的规模处理程序


2. 使用实例组编写一个简单的自动缩放处理程序，然后使用 InstanceGroups：insert


3. 使用 InstangeGroupManagers：resize


4. 创建一个自定义的实例度量标准，它对所有工作人员采取措施 len（队列）/ len（工人）
   

解决方案

据我了解，这尚未实施（截至2016年1月）。目前，自动缩放仅针对Web服务场景，您希望从您的计算机为网页/其他Web服务提供服务，并为流量尖峰保留一些合理的空间（例如，在CPU或其他指标方面）。然后，系统会调整实例/虚拟机的数量以匹配您的目标。

您正在寻找批量处理场景的自动缩放，目前尚不满足。

We host our infrastructure on Google Compute Engine and are looking into Autoscaling for groups of instances. We do a lot of batch processing of binary data from a queue. In our case, this means:

1. When a worker is processing data the CPU is always 100%
2. When the queue is empty we want to terminate all workers
3. Depending on the length of the queue we want a certain amount of workers

However I'm finding it hard to figure out a way to auto-scale this on Google Compute Engine because they appear to scale on instance-only metrics such as CPU. From the documentation:

Not all custom metrics can be used by the autoscaler. To choose a valid custom metric, the metric must have all of the following properties:

• The metric must be a per-instance metric.
• The metric must be a valid utilization metric, which means that data from the metric can be used to proportionally scale up or down the number of virtual machines.

If I'm reading the documentation properly this makes it hard to use the auto scaling on a global queue length?

Backup solutions

1. Write a simple auto-scale handler using the Google Cloud API to create or destroy new workers using Instances API
2. Write a simple auto-scale handler using instance groups and then manually insert/remove instances using the InstanceGroups: insert
3. Write a simple auto-scaling handler using InstangeGroupManagers: resize
4. Create a custom per-instance metric which measures len(queue)/len(workers) on all workers

解决方案

As far as I understand this is not implemented yet (as at January 2016). At the moment autoscaling is only targeted at web serving scenarios, where you want to serve web pages/other web services from your machines and keep some reasonable headroom (e.g. in terms of CPU or other metrics) for spikes in traffic. Then the system will adjust the number of instances/VMs to match your target.

You are looking for autoscaling for batch processing scenarios, and this is not catered for at the moment.

这篇关于Google Compute Engine根据队列长度自动缩放的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！