Cron 作业中的 Scrapy 爬虫 [英] Scrapy crawler in Cron job

问题描述

我想从 cron 作业执行我的爬虫爬虫.

I want to execute my scrapy crawler from cron job .

我创建了 bash 文件 getdata.sh，scrapy 项目所在的位置有它的蜘蛛

i create bash file getdata.sh where scrapy project is located with it's spiders

#!/bin/bash
cd /myfolder/crawlers/
scrapy crawl my_spider_name

我的crontab是这样的，我想每5分钟执行一次

My crontab looks like this , I want to execute it in every 5 minute

 */5 * * * * sh /myfolder/crawlers/getdata.sh 

但它不起作用，怎么了，我的错误在哪里?

but it don't works , whats wrong , where is my error ?

当我从终端 sh/myfolder/crawlers/getdata.sh 执行我的 bash 文件时它工作正常

when I execute my bash file from terminal sh /myfolder/crawlers/getdata.sh it works fine

推荐答案

我解决了这个问题，包括将PATH导入bash文件

I solved this problem including PATH into bash file

#!/bin/bash

cd /myfolder/crawlers/
PATH=$PATH:/usr/local/bin
export PATH
scrapy crawl my_spider_name

这篇关于Cron 作业中的 Scrapy 爬虫的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！