使用scrapyd运行多个蜘蛛 [英] Running multiple spiders using scrapyd

问题描述

我的项目中有多个蜘蛛，所以决定通过上传到scrapyd服务器来运行它们.我已经成功上传了我的项目，当我运行命令时，我可以看到所有的蜘蛛

I had multiple spiders in my project so decided to run them by uploading to scrapyd server. I had uploaded my project succesfully and i can see all the spiders when i run the command

curl http://localhost:6800/listspiders.json?project=myproject

当我运行以下命令时

curl http://localhost:6800/schedule.json -d project=myproject -d spider=spider2

因为只给出了一个蜘蛛，所以只有一个蜘蛛运行，但我想在这里运行多个蜘蛛，所以下面的命令适合在scrapyd中运行多个蜘蛛?

Only one spider runs because of only one spider given, but i want to run run multiple spiders here so the following command is right for running multiple spiders in scrapyd ?

curl http://localhost:6800/schedule.json -d project=myproject -d spider=spider1,spider2,spider3........

稍后我将使用 cron 作业运行此命令，我的意思是我将安排它频繁运行

And later i will run this command using cron job i mean i will schedule this to run frequently

推荐答案

如果你想使用scrapyd运行多个蜘蛛，请一个一个地调度它们.scrapyd 会以相同的顺序运行它们，但不会同时运行.

If you want to run multiple spiders using scrapyd, schedule them one by one. scrapyd will run them in the same order but not at the same time.

另见:Scrapy 的 Scrapyd 因调度蜘蛛太慢

这篇关于使用scrapyd运行多个蜘蛛的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！