Apache Beam:PubsubReader 因 NPE 失败 [英] Apache Beam: PubsubReader fails with NPE

问题描述

我有一个光束管道，它在应用一些转换后从 PubSub 读取并写入 BigQuery.管道始终以 NPE 失败.我正在使用梁 SDK 版本 0.6.0.关于我可能做错了什么的任何想法?我正在尝试使用 DirectRunner 运行管道.

I have a a beam pipeline that reads from PubSub and write to BigQuery after applying some transformation. The pipeline fails consistently with a NPE. I am using beam SDK version 0.6.0. Any Idea on what I could be doing wrong? I am trying to run the pipeline with a DirectRunner.

java.lang.NullPointerException
at org.apache.beam.sdk.io.PubsubUnboundedSource$PubsubReader.ackBatch(PubsubUnboundedSource.java:640)
at org.apache.beam.sdk.io.PubsubUnboundedSource$PubsubCheckpoint.finalizeCheckpoint(PubsubUnboundedSource.java:313)
at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.getReader(UnboundedReadEvaluatorFactory.java:174)
at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.processElement(UnboundedReadEvaluatorFactory.java:127)
at org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
at org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

推荐答案

这个问题存在是因为一个 Bug (BEAM-1656)和 PubsubCheckpoint 中的前提条件.DirectRunner 中的 bug 已在 pull request 2237 中修复，已合并到 Githubmaster 分支，但在 0.6.0 发布之后.

This problem exists because of a Bug (BEAM-1656) in the DirectRunner and a precondition within PubsubCheckpoint. The bug in the DirectRunner was fixed in pull request 2237, which is merged into the Github master branch, but after 0.6.0 release.

更新到 0.7.0 nightly build 或从 github HEAD 构建将在使用 DirectRunner 时解决此问题.

Updating to the 0.7.0 nightly build or building from github HEAD will solve this problem when using the DirectRunner.

要更新到当前的每晚构建，您必须将以下存储库添加到项目的 pom.xml.包含修复程序的 beam-runners-direct-java 模块的最早版本是 0.7.0-20170316.070901-9，但并非所有模块都使用此特定版本构建，因此您可能必须指定单独兼容的版本或使用 0.7.0-SNAPSHOT

To update to the current nightly build you will have to add the following repositories to your project's pom.xml. The earliest version of the beam-runners-direct-java module containing the fix is 0.7.0-20170316.070901-9, but not all modules are built with this specific version so you may have to either specify individually compatible versions or use 0.7.0-SNAPSHOT

    <repositories>
      <repository>
        <id>apache.snapshots</id>
        <name>Apache Development Snapshot Repository</name>

 <url>https://repository.apache.org/content/repositories/snapshots/</url>
        <releases>
          <enabled>false</enabled>
        </releases>
        <snapshots>
          <enabled>true</enabled>
        </snapshots>
      </repository>

    </repositories>

这篇关于Apache Beam:PubsubReader 因 NPE 失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！