我如何使用 wget 下载带有scrapy的文件 [英] how can i use wget to download files with scrapy

问题描述

scrapy 没有正确下载文件.我有我的项目的 URL，所以我想我可以使用 wget 下载文件.

scrapy is not downloading files properly. I have URLs of my items, so I figured I can use wget to download the files.

如何在scrapy process_item 函数中使用wget?或者，还有其他下载文件的方式吗?

How can i use wget inside the scrapy process_item function? Alternatively, is there another way of download files?

class MyImagesPipeline(ImagesPipeline):
    #Name download version
    def image_key(self, url):
        image_guid = url.split('/')[-1]
        return 'full/%s' % (image_guid)

    def get_media_requests(self, item, info):
        if item['image_urls']:
            for image_url in item['image_urls']:
            # wget -nH image_ul -P  images/
            yield Request(image_url)

推荐答案

这段代码会执行wget，你可以用下面几行替换你的评论

This code will execute wget, you can replace your comment with the following lines

import subprocess

    ...

subprocess.call(['wget','-nH', image_url, '-P  images/'])

您可以在此处阅读有关 subprocess.call 的信息:http://docs.python.org/2/library/subprocess.html

You can read about subprocess.call here: http://docs.python.org/2/library/subprocess.html

这篇关于我如何使用 wget 下载带有scrapy的文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！