用于从 Storm Spout 中读取 Kestrel 服务器队列的 Java 客户端库 [英] Java client library for reading from Kestrel server Queue from within Storm Spout

问题描述

我已经设置了 Kestrel 服务器，并且能够通过 python pykestrel 库设置和使用队列.

I've setup a Kestrel server and able to setup and use queues via the python pykestrel library.

我们有一个场景，python 客户端写入 Kestrel 队列，而 Storm spout 需要从队列中读取.我试过使用storm-kestrel 库，但遇到了问题.谷歌搜索似乎表明它不支持内存缓存端口 (22133).

We have a scenario where a python client writes to Kestrel queue(s) and a Storm spout needs to read from the queue(s). I've tried using the storm-kestrel library but running into issues. Googling seems to suggest it doesn't support the memcache port (22133).

我已按照此处的规定添加了 Maven 绑定.没有使用 KestrelThriftSpout 喷口，而是使用 Kestrel.Client.编译没问题，但在运行时出现 java.lang.NoClassDefFoundError 错误.

I've added the maven bindings as provided here. Didn't use the KestrelThriftSpout spout, using Kestrel.Client. Compilation is fine but I get java.lang.NoClassDefFoundError error at runtime.

堆栈跟踪

1098 [main] ERROR org.apache.zookeeper.server.NIOServerCnxn  - Thread Thread[main,5,main] died
java.lang.NoClassDefFoundError: Lnet/lag/kestrel/thrift/Kestrel$Client;
    at java.lang.Class.getDeclaredFields0(Native Method)
    at java.lang.Class.privateGetDeclaredFields(Class.java:2397)
    at java.lang.Class.getDeclaredField(Class.java:1946)
    at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1659)
    at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)
    at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:480)
    at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)
    at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
    at backtype.storm.utils.Utils.serialize(Utils.java:52)
    at backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:94)
    at com.rr.storm.ArticleTolopolgy.main(ArticleTolopolgy.java:35)
Caused by: java.lang.ClassNotFoundException: net.lag.kestrel.thrift.Kestrel$Client
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 15 more

有人可以确认在这种情况下是否可以使用storm-kestrel吗?

Can someone please confirm if storm-kestrel can or cannot be used in this scenario?

如果没有，是否有另一个 kestrel 客户端库推荐在风暴喷口内使用，以从 kestrel 队列读取数据?

If not, is there another kestrel client library that is recommended to be used from within a storm spout for reading data off of kestrel queue(s)?

推荐答案

抱歉更新晚了.

• java.lang.NoClassDefFoundError 是因为某些 JAR 不在类路径中.涉及到几个依赖项，因此我们通过将所有依赖项打包到单个 JAR 中来解决该问题.有几个依赖冲突需要解决，尤其是.一个令人讨厌的涉及 AWS SQS 和风暴取决于不同的 httpclient 库.在发布时需要将 AWS SQS Java SDK 版本降级到 1.4.7 而不是最新的 1.7.2.

• The java.lang.NoClassDefFoundError was because some JARs were not in classpath. There were several dependencies involved so we resolved the issue by packing all dependencies into a SINGLE JAR. There were several dependency conflicts that needed to be resolved esp. a nasty one involving AWS SQS and storm depending on different httpclient libraries. Required downgrading the AWS SQS Java SDK version to 1.4.7 instead of the latest 1.7.2 at the time of posting.

它看起来不像 storm-kestrel 库与 kestrel 一起使用服务器不再存在，但如果理解错误，其他人可以纠正.

It doesn't look like the storm-kestrel library works with the kestrel server anymore but someone else can correct if the understanding is wrong.

将 pykestrel 用于 Python 客户端，效果很好.

Used pykestrel for Python client and it works great.

将 simple-kestrel-client 用于 Java kestrel 客户端(Storm spout) 并且效果很好

Used simple-kestrel-client for Java kestrel client (Storm spout) and it works great

使用的 Java SDK v1.4.7 由于冲突而适用于 AWS SQS，并且效果很好.

Used Java SDK v1.4.7 for AWS SQS due to the conflict and it works great.

这篇关于用于从 Storm Spout 中读取 Kestrel 服务器队列的 Java 客户端库的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！