Apache Beam:PubsubReader失败并带有NPE [英] Apache Beam: PubsubReader fails with NPE

问题描述

我有一个束流管道，该束流从PubSub读取，并在应用了一些转换后写入BigQuery.管道与NPE一致失败.我正在使用Beam SDK版本0.6.0.对我可能做错的任何主意吗?我正在尝试使用DirectRunner运行管道.

I have a a beam pipeline that reads from PubSub and write to BigQuery after applying some transformation. The pipeline fails consistently with a NPE. I am using beam SDK version 0.6.0. Any Idea on what I could be doing wrong? I am trying to run the pipeline with a DirectRunner.

java.lang.NullPointerException
at org.apache.beam.sdk.io.PubsubUnboundedSource$PubsubReader.ackBatch(PubsubUnboundedSource.java:640)
at org.apache.beam.sdk.io.PubsubUnboundedSource$PubsubCheckpoint.finalizeCheckpoint(PubsubUnboundedSource.java:313)
at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.getReader(UnboundedReadEvaluatorFactory.java:174)
at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.processElement(UnboundedReadEvaluatorFactory.java:127)
at org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
at org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

推荐答案

由于存在错误( 2237 中得到修复，该请求已合并到Github中主分支，但在0.6.0版本发布之后.

This problem exists because of a Bug (BEAM-1656) in the DirectRunner and a precondition within PubsubCheckpoint. The bug in the DirectRunner was fixed in pull request 2237, which is merged into the Github master branch, but after 0.6.0 release.

使用DirectRunner时，将夜间版本更新为0.7.0或从github HEAD进行构建将解决此问题.

Updating to the 0.7.0 nightly build or building from github HEAD will solve this problem when using the DirectRunner.

要更新到当前的夜间版本，您必须将以下存储库添加到项目的pom.xml中.包含此修复程序的beam-runners-direct-java模块的最早版本是0.7.0-20170316.070901-9，但是并非所有模块都使用此特定版本构建，因此您可能必须指定单独兼容的版本或使用0.7.0-SNAPSHOT

To update to the current nightly build you will have to add the following repositories to your project's pom.xml. The earliest version of the beam-runners-direct-java module containing the fix is 0.7.0-20170316.070901-9, but not all modules are built with this specific version so you may have to either specify individually compatible versions or use 0.7.0-SNAPSHOT

    <repositories>
      <repository>
        <id>apache.snapshots</id>
        <name>Apache Development Snapshot Repository</name>

 <url>https://repository.apache.org/content/repositories/snapshots/</url>
        <releases>
          <enabled>false</enabled>
        </releases>
        <snapshots>
          <enabled>true</enabled>
        </snapshots>
      </repository>

    </repositories>

这篇关于Apache Beam:PubsubReader失败并带有NPE的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！