以这个比例查询耗尽的资源 [英] Query exhausted resources at this scale factor

问题描述

我正在Amazon Athena上运行SQL查询。而且我几次出现以下错误：

I was running SQL query on Amazon Athena. And I got the following error couple of times:

以这种比例系数查询耗尽的资源

Query exhausted resources at this scale factor

此查询已运行针对 test1数据库，除非由查询限定。请在我们的论坛上发布错误消息，或通过查询ID与客户支持联系：*************

This query ran against the "test1" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: *************

推荐答案

没有看到查询，很难确定是什么问题，但是很可能是由于Athena的内部问题，该问题与大型中间结果集的排序有关。

Without seeing the query it's hard to say for sure what the problem is, but it's very likely that it is due to an internal issue in Athena that has to do with sorting of large intermediary result sets.

Athena使用的Presto版本不支持对太大而无法容纳在内存中的数据集进行排序。

The version of Presto that Athena uses does not have support for sorting datasets that are too big to fit in memory. It used to be the same for aggregations too, but that has been fixed by the Athena team.

该问题通常发生在表非常宽（即多列）的情况下，或包含大量数据的列。每行都可以代表很大的内存块，如果节点在尝试对其内存块进行排序时用完了内存，则查询将中止，并显示查询耗尽资源（按此比例因子）错误。

The issue most often occurs when you have very wide tables, i.e. many columns, or columns with a lot of data. Each individual row can represent a big chunk of memory, and if a node runs out of memory while trying to sort its chunk the query aborts with the "query exhausted resources at this scale factor" error.

如果这符合您的情况，那么不幸的是，唯一的方法是限制列数或取消排序。有时，您可以重新安排查询以在另一个阶段进行排序，以降低排序阶段的内存压力。

If this matches your situation the only way around is unfortunately to limit the number of columns, or eliminate the sorting. Sometimes you can rearrange the query to do the sorting at a different stage to make the memory pressure on the sorting stage lower.

这篇关于以这个比例查询耗尽的资源的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！