BigQuery UDF 内部错误 [英] BigQuery UDF Internal Error

问题描述

我们在 BigQuery 中有一个简单的 UDF，它以某种方式抛出一个不断返回的错误

We had a simple UDF in BigQuery that somehow throws an error that keeps returning

Query Failed
Error: An internal error occurred and the request could not be completed.

该查询只是尝试使用 UDF 来执行 SHA256.

The query was simply trying to use UDF to perform a SHA256.

SELECT
  input AS title,
  input_sha256 AS title_sha256
FROM
  SHA256(
      SELECT
        title AS input
      FROM
        [bigquery-public-data:hacker_news.stories]
      GROUP BY
        input 
  )
LIMIT
  1000

内嵌 UDF 粘贴在下面.但是我不能发布完整的 UDF，因为 StackOverflow 抱怨帖子中的代码太多.完整的 UDF 可以在这个要点中看到.

The in-line UDF is pasted below. However I can not post the full UDF as StackOverflow complaints too much code in the post. The full UDF can be seen this gist.

function sha256(row, emit) {
  emit(
      {
        input: row.input,
        input_sha256: CryptoJS.SHA256(row.input).toString(CryptoJS.enc.Hex)
      }
  );
}

bigquery.defineFunction(
  'SHA256',                           // Name of the function exported to SQL
  ['input'],                    // Names of input columns
  [
      {'name': 'input', 'type': 'string'},
      {'name': 'input_sha256', 'type': 'string'}
  ],
  sha256                       // Reference to JavaScript UDF
);

不确定是否有帮助，但 Job-ID 是

Not sure if it helps, but the Job-ID is

bigquery:bquijob_7fd3b51c_153c058dc7c

看起来有类似的问题:

https://code.google.com/p/google-bigquery/issues/detail?id=478

推荐答案

Short answer - 这是一个与内存分配相关的问题，我通过自己的测试发现并在今天修复，但需要一段时间才能流出生产.

Short answer - this is an issue related to memory allocation that I uncovered via my own testing and fixed today, but it will take a little while to flow out to production.

稍微长一点的答案 - 我们今天刚刚推出了一个修复程序，解决了用户在将 UDF 扩展到更多行时遇到内存不足"问题的问题，即使 UDF 会在更少的行上成功.达到该条件的查询现在在我们的内部/测试树上运行良好.但是，由于公共 BigQuery 主机的流量负载要高得多，因此执行 UDF (V8) 的 JavaScript 引擎在生产中的行为与在内部树中的行为略有不同.具体来说，有一个新的内存分配错误，一些以前的 OOM 作业现在正在命中，直到查询在完全加载的树上运行时我们才能观察到.

Slightly longer answer - we just rolled out a fix today for an issue where users who were having "out of memory" issues when scaling up their UDFs over larger number of rows, even though the UDF would succeed on smaller numbers of rows. The queries that were hitting that condition are now running fine on our internal / test trees. However, since public BigQuery hosts have much higher traffic loads, the JavaScript engine that executes the UDFs (V8) behaves somewhat differently in production than it does in internal trees. Specifically, there's a new memory allocation error that some of the previously OOMing jobs are now hitting that we couldn't observe until the queries ran on a fully-loaded tree.

这是一个可以快速修复的小错误，但我们最好让它通过我们的常规测试和 QA 周期.假设候选人没有其他问题，这应该会在大约一周内将修复程序投入生产.你能接受吗?

It's a minor error with a quick fix, but we'd ideally let it flow through our regular testing and QA cycle. This should put the fix in production in about a week, assuming nothing else goes wrong with the candidate. Would that be acceptable for you?

这篇关于BigQuery UDF 内部错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！