在 Dataflow 中运行 Apache Beam Pipeline 时出现 SSLHandshakeException [英] SSLHandshakeException when running Apache Beam Pipeline in Dataflow
本文介绍了在 Dataflow 中运行 Apache Beam Pipeline 时出现 SSLHandshakeException的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我有一个 Apache Beam 管道.在 DoFn 步骤之一中，它执行 https 调用(想想 REST API).所有这些都可以在我的本地环境中使用 DirectRun 正常工作.
这是我本地环境，apache beam 2.16.0:
$ mvn -versionApache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-04T12:00:29-07:00)Maven 主页:/opt/apache-maven-3.6.1Java 版本:1.8.0_222，供应商:私有构建，运行时:/usr/lib/jvm/java-8-openjdk-amd64/jre默认语言环境:en，平台编码:UTF-8操作系统名称:linux"，版本:5.0.0-1025-gcp"，arch:amd64"，系列:unix"但是，当我部署到 Dataflow 时，一切正常，只是 https 调用失败并显示如下所示的堆栈跟踪.
有关如何解决和克服此问题的任何建议?
javax.net.ssl.SSLHandshakeException:收到致命警报:handshake_failure在 sun.security.ssl.Alerts.getSSLException(Alerts.java:192)在 sun.security.ssl.Alerts.getSSLException(Alerts.java:154)在 sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2033)在 sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135)在 sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)在 sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)在 sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)在 okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:351)在 okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:310)在 okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:178)在 okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:236)在 okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:109)在 okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:77)在 okhttp3.internal.connection.Transmitter.newExchange$okhttp(Transmitter.kt:162)在 okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:35)在 okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)在 okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)在 okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:82)在 okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)在 okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)在 okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:84)在 okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)在 okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:71)在 okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)在 okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)在 okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.kt:184)在 okhttp3.RealCall.execute(RealCall.kt:66)...在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)在 org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:335)在 org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)在 org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)在 org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:280)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:252)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:74)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:576)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:564)...在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)在 org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:335)在 org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)在 org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)在 org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:280)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:252)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:74)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:576)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:564)...在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)在 org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:335)在 org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)在 org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)在 org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:280)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:252)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:74)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:576)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:564)...在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)在 org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:335)在 org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)在 org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)在 org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:280)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:252)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:74)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:576)在 org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:71)在 org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:139)在 org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(来源不明)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)在 org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)在 org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:335)在 org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)在 org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)在 org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)在 org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)在 org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)在 org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1316)在 org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:149)在 org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1049)在 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)在 java.lang.Thread.run(Thread.java:745)时间戳2019-12-01T18:11:22.370Z记录器严重性警告工人步线524我按照
解决方案
TLDR
从 Beam 2.14 开始，您可以创建将在 JVM 实例化上运行的 JvmInitializer 的实现.实施并注册一个以覆盖谷歌的安全设置.
@AutoService(value = JvmInitializer.class)公共类 MyJvmInitializer 实现 JvmInitializer {@覆盖公共无效 onStartup() {Security.setProperty("jdk.tls.disabledAlgorithms", "");}@覆盖public void beforeProcessing(PipelineOptions options) {}}要注册上述内容，我使用 AutoService，如果您愿意，您可以进行不同的注册.
说明
从强大的密码算法来看，Java仅支持GCM，尚不支持CHACHA20_POLY1305.在您的本地/DirectRunner 中，JVM 使用默认设置进行实例化，其中包括 GCM 算法.所以它可以使用服务器支持的 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 或 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384.
另一方面，Google Dataflow 将带有安全设置的属性文件传递给工作线程 JVM.属性文件位于 /opt/google/dataflow/tls/disable_gcm.properties 中的工作器虚拟机中.
这些属性明确禁用 GSM 算法.文件内容为
# 我们明确禁用 GCM，因为 Java 8 有一个简单的和# 不切实际地缓慢实现它.## 由于 POODLE 漏洞，JDK 中默认禁用 SSLv3.# 有关更多详细信息，请参阅 CVE-2014-3566.jdk.tls.disabledAlgorithms=SSLv3，GCM# 由于缺乏虚拟化环境中的/dev/random 块# 良好的熵源，这使得 SecureRandom 的使用不切实际.# 特别是，这会影响依赖的 HTTPS 的性能# on SecureRandom.## 因此，/dev/urandom 被用作默认值.## 一些背景见 http://www.2uo.de/myths-about-urandom/# 关于 Linux 上/dev/urandom 的安全性.securerandom.source=file:/dev/./urandom"；因此，Dataflow 工作器中的 HTTP 客户端与您尝试访问的服务器之间没有通用密码.
I have an Apache Beam Pipeline. In one of the DoFn steps it does an https call (think REST API). All this works fine with DirectRun in my local environment.
This is my local environment, apache beam 2.16.0:
$ mvn -version
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-04T12:00:29-07:00)
Maven home: /opt/apache-maven-3.6.1
Java version: 1.8.0_222, vendor: Private Build, runtime: /usr/lib/jvm/java-8-openjdk-amd64/jre
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "5.0.0-1025-gcp", arch: "amd64", family: "unix"
However, when I deploy to Dataflow everything works fine except that the https calls fail with the stack trace shown below.
Any suggestion on how to troubleshoot and overcome this issue?
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2033)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:351)
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:310)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:178)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:236)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:109)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:77)
at okhttp3.internal.connection.Transmitter.newExchange$okhttp(Transmitter.kt:162)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:35)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:82)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:84)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:71)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.kt:184)
at okhttp3.RealCall.execute(RealCall.kt:66)
...
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:335)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:280)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:252)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:74)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:576)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:564)
...
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:335)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:280)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:252)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:74)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:576)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:564)
...
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:335)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:280)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:252)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:74)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:576)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:564)
...
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:335)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:280)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:252)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:74)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:576)
at org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:71)
at org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:139)
at org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown Source)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:335)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1316)
at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:149)
at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1049)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
timestamp
2019-12-01T18:11:22.370Z
logger
severity
WARNING
worker
step
thread
524
I followed the directions in this thread and limited the CipherSuites to the following:
ConnectionSpec spec = new ConnectionSpec.Builder(ConnectionSpec.COMPATIBLE_TLS)
.tlsVersions(TlsVersion.TLS_1_3, TlsVersion.TLS_1_2)
.cipherSuites(
CipherSuite.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384)
.build();
httpClient = new OkHttpClient.Builder()
.connectionSpecs(Collections.singletonList(spec))
.build();
Again it works fine with DirectRun, when submitted to DataFlow the error now is a bit more specific:
java.net.UnknownServiceException: Unable to find acceptable protocols. isFallback=false, modes=[ConnectionSpec(cipherSuites=[TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384], tlsVersions=[TLS_1_3, TLS_1_2], supportsTlsExtensions=true)], supported protocols=[TLSv1, TLSv1.1, TLSv1.2]
at okhttp3.internal.connection.ConnectionSpecSelector.configureSecureSocket(ConnectionSpecSelector.kt:63)
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:345)
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:310)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:178)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:236)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:109)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:77)
at okhttp3.internal.connection.Transmitter.newExchange$okhttp(Transmitter.kt:162)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:35)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:82)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:84)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:71)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.kt:184)
at okhttp3.RealCall.execute(RealCall.kt:66)
...
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:335)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:280)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:252)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:74)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:576)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:564)
...
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:335)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:280)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:252)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:74)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:576)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:564)
...
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:335)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:280)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:252)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:74)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:576)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:564)
...
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:335)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:280)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:252)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:74)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:576)
at org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:71)
at org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:139)
at org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown Source)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:335)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1316)
at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:149)
at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1049)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Server SSL Report:
解决方案
TLDR
Since Beam 2.14, you can create an implementation of a JvmInitializer that will run on JVM instantiation. Implement and register one to override google's security settings.
@AutoService(value = JvmInitializer.class)
public class MyJvmInitializer implements JvmInitializer {
@Override
public void onStartup() {
Security.setProperty("jdk.tls.disabledAlgorithms", "");
}
@Override
public void beforeProcessing(PipelineOptions options) {}
}
To register the above I use AutoService, you can register it differently if you prefer.
Explanation
From the strong algorithms for ciphers, only GCM is supported in Java, CHACHA20_POLY1305 is not supported yet. In your local/DirectRunner, the JVM is instantiated with the default settings, which include the GCM algos. So it can use TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 or TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 which are both supported by the server.
Google Dataflow, on the other hand, passes a property file with security settings to the worker JVM. The property file is in the worker VM in /opt/google/dataflow/tls/disable_gcm.properties.
These properties explicitly disable GSM algos. The content of the file is
# We disable GCM explicitly, since Java 8 has a naive and
# impractically slow implementation of it.
#
# SSLv3 is disabled in JDK by default due to POODLE vulnerability.
# See CVE-2014-3566 for more details.
jdk.tls.disabledAlgorithms=SSLv3, GCM
# /dev/random blocks in virtualized environments due to lack of
# good entropy sources, which makes SecureRandom use impractical.
# In particular, that affects the performance of HTTPS that relies
# on SecureRandom.## Due to that, /dev/urandom is used as the default.
#
# See http://www.2uo.de/myths-about-urandom/ for some background
# on security of /dev/urandom on Linux.
securerandom.source=file:/dev/./urandom"
So, there are no common ciphers between an HTTP client in the Dataflow workers and the server you are trying to reach.
这篇关于在 Dataflow 中运行 Apache Beam Pipeline 时出现 SSLHandshakeException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
