如何在分布式环境中运行 nutch 服务器 [英] How to run nutch server on distributed environment

问题描述

我已经通过使用 bin/nutch startserver 命令本地启动它来测试在服务器模式下运行 nutch.现在我想知道我是否可以在 hadoop 集群之上(在分布式环境中)以 服务器模式 启动 nutch 并使用 nutch REST api 向服务器提交抓取请求?请帮忙.

I have tested running of nutch in server mode by starting it using bin/nutch startserver command locally. Now I wonder whether I can start nutch in server mode on top of a hadoop cluster(in distributed environment) and submit crawl requests to server using nutch REST api ? Please help.

推荐答案

通过进一步的研究，我让 nutch 服务器在分布式模式下工作.

From further research I've got nutch server working on distributed mode.

步骤:-

1. 假设所有slave节点都配置了hadoop.然后在所有节点中设置 nutch.这可以帮助:http://wiki.apache.org/nutch/NutchHadoopTutorial
2. 在您的名称节点上，cd $NUTCH_HOME/runtime/deploy
3. bin/nutch startserver -port <端口>-host 注意:端口和主机是可选的.
4. 然后您可以使用 REST 从 nutch 提交请求.您提交的请求将被第 3 步启动的 nutch 服务器接受.

1. Assume hadoop is configured in all slave nodes. Then setup nutch in all nodes. This can help : http://wiki.apache.org/nutch/NutchHadoopTutorial
2. On your namenode, cd $NUTCH_HOME/runtime/deploy
3. bin/nutch startserver -port <port> -host <host> Note :Port and host are optional.
4. Then you can submit requests from nutch using REST. The requests you submit will be accepted by nutch server started on step 3.

快乐爬行:)

这篇关于如何在分布式环境中运行 nutch 服务器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！