在本地模式下运行 StormCrawler 还是安装 Apache Storm? [英] Run StormCrawler in local mode or install Apache Storm?

问题描述

所以我想弄清楚如何使用 ES 和 Kibana 安装和设置 Storm/Stormcrawler，如 此处.

So I'm trying to figure out how to install and setup Storm/Stormcrawler with ES and Kibana as described here.

我从来没有在我的本地机器上安装过 Storm，因为我以前用过 Nutch，而且我从来没有在本地安装过 Hadoop……我认为 Storm 可能也一样(也许不是?).

I never installed Storm on my local machine because I've worked with Nutch before and I never had to install Hadoop locally... thought it might be the same with Storm(maybe not?).

我现在想开始用 Stormcrawler 而不是 Nutch 爬行.

I'd like to start crawling with Stormcrawler instead of Nutch now.

看来，如果我只是下载一个版本并将/bin 添加到我的 PATH 中，我可以只与远程集群通信.

It seems that if I just download a release and add the /bin to my PATH, I can only talk to a remote cluster.

看来我需要根据 this，让我能够随着时间的推移开发不同的拓扑，然后在准备部署新拓扑时从我的本地机器与远程集群交谈.是吗?

It seems like I need to setup a development environment according to this, to give me the ability to develop different topologies over time and then just talk to the remote cluster from my local machine when ready to deploy the new topologies. Is that right?

所以似乎我需要做的就是在我使用 Maven 构建它时将 Storm 作为依赖项添加到我的 Stormcrawler 项目中?

So it seems like all I need to do is add Storm as a dependency to my Stormcrawler project when I build it with Maven?

推荐答案

请参阅入门页面以及 Youtube 上的教程.

您无需安装 Storm，因为您可以在本地模式下运行拓扑，就像使用 Nutch 和 Hadoop 一样.只需从原型生成拓扑，根据您的需要修改它，例如添加 ES 组件并使用 -local 运行它.参见原型生成的 README.

You don't need to install Storm as you can run the topology in local mode, just as you'd do with Nutch and Hadoop. Just generate a topology from the archetype, modify it to your needs e.g. add ES components and run it with -local. See README generated by the archetype.

稍后，您将安装 Storm 以从 UI 中受益，并可能在多个节点上运行它，但作为起点，在本地执行此操作是探索 StormCrawler 功能的好方法.

Later on, you'd install Storm to benefit from the UI and possibly run it on multiple nodes but as a starting point doing it locally is a good way of exploring the capabilities of StormCrawler.

这篇关于在本地模式下运行 StormCrawler 还是安装 Apache Storm?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！