将该URL传递给从RabbitMQ消耗的scrapy中的parse方法 [英] Pass the url into the parse method in scrapy that was consumed from RabbitMQ

问题描述

我正在使用scrapy消费RabbitMQ的message(url)，但是当我使用yield调用将URL作为参数传递给parse方法时，程序不包含在回调方法中.我的蜘蛛的代码

I am using the scrapy to consume the message(url) from the RabbitMQ,But When I use the yield to call the parse method passing my url as parameters .The program does not comes inside the callback method.Below is the foloowing code of my spider

# -*- coding: utf-8 -*-
import scrapy
import pika
from scrapy import cmdline
import json

class MydeletespiderSpider(scrapy.Spider):
    name = 'Mydeletespider'
    allowed_domains = []
    start_urls = []

def callback(self,ch, method, properties, body):
    print(" [x] Received %r" % body)
    body=json.loads(body)
    url=body.get('url')
    yield scrapy.Request(url=url,callback=self.parse)

def start_requests(self):
    cre = pika.PlainCredentials('test', 'test')
    connection = pika.BlockingConnection(
        pika.ConnectionParameters(host='10.0.12.103', port=5672, credentials=cre, socket_timeout=60))
    channel = connection.channel()



    channel.basic_consume(self.callback,
                          queue='Deletespider_Batch_Test',
                          no_ack=True)


    print(' [*] Waiting for messages. To exit press CTRL+C')
    channel.start_consuming()

def parse(self, response):
    print response.url
    pass

cmdline.execute('scrapy crawl Mydeletespider'.split())

我的目标是将url响应传递给解析方法

My goal is to pass the url response to parse method

推荐答案

请参阅: http://30daydo .com/article/512

def start_requests(self)，此函数应返回一个生成器，否则将无法正常工作.

def start_requests(self) this function should return a generator, else scrapy wont work.

这篇关于将该URL传递给从RabbitMQ消耗的scrapy中的parse方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！