将文本文件写入管道 [英] Write text file to pipeline

问题描述

我在一个scrapy项目中有多个蜘蛛.

I have multiple spiders in a single scrapy project.

我想为每个带有蜘蛛名称和时间戳的蜘蛛编写一个单独的输出文本文件.

I want to write a separate output text file for each spider with spider name and time stamp.

当我有一个蜘蛛时，我在 __init 方法中创建文件，但现在我正在尝试这样做，upromise 将生成两个输出文件，而其他只会生成一个.

When I had a single spider I was creating file in __init method but now I am trying like this, upromise will generate two output files while other will only one.

class MallCrawlerPipeline(object):

def spider_opened(self, spider):
    self.aWriter = csv.writer(open('../%s_%s.txt' % (spider.name, datetime.now().strftime("%Y%m%d_%H%M%S")), 'wb'),
        delimiter=',', quoting=csv.QUOTE_MINIMAL)
    self.aWriter.writerow(['mall', 'store', 'bonus', 'per_action', 'more_than','up_to', 'deal_url', 'category'])

    if 'upromise' in spider.name:
        self.cWriter = csv.writer(
            open('../%s_coupons_%s.txt' % (spider.name, datetime.now().strftime("%Y%m%d_%H%M%S")), 'wb'),
            delimiter=',', quoting=csv.QUOTE_MINIMAL)
        self.cWriter.writerow(['mall', 'store', 'bonus', 'per_action', 'more_than','up_to', 'deal_url', 'category'])

def process_item(self, item, spider):
    self.aWriter.writerow([item['mall'], item['store'], item['bonus'], item['per_action'],
                           item['more_than'], item['up_to'], item['deal_url'], item['category']])

    return item

但我正面临这个错误:

 File "C:\Users\akhter\Dropbox\akhter\mall_crawler\mall_crawler\pipelines.py", line 24, in process_item
    self.aWriter.writerow([item['mall'], item['store'], item['bonus'], item['per_action'],
exceptions.AttributeError: 'MallCrawlerPipeline' object has no attribute 'aWriter'

任何帮助将不胜感激.提前致谢.

Any help would be appreciated. Thanks in advance.

推荐答案

谢谢大家，我找到了一个答案，我只需要给出一个信号，否则 Spider_opened 永远不会在 init 方法中像这样调用.仍然开放接受建议

thanks guys i found an answer i just need to give a signal otherwise spider_opened never call like this in init method. still open for suggestions

def __init__(self):
    dispatcher.connect(self.spider_opened, signals.spider_opened)

这篇关于将文本文件写入管道的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！