multiprocessing.pool.MaybeEncodingError:发送结果错误:原因:'TypeError(“无法序列化'_io.BufferedReader'object"，)' [英] multiprocessing.pool.MaybeEncodingError: Error sending result: Reason: 'TypeError("cannot serialize '_io.BufferedReader' object",)'

问题描述

我收到以下错误:

multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f758760d6a0>'. Reason: 'TypeError("cannot serialize '_io.BufferedReader' object",)'

运行此代码时:

from operator import itemgetter
from multiprocessing import Pool
import wget


def f(args):
    print(args[1])
    wget.download(args[1], "tests/" + target + '/' + str(args[0]), bar=None)

if __name__ == "__main__":
    a = Pool(2)
    a.map(f, list(enumerate(urls))) #urls is a list of urls.

错误是什么意思，我该如何解决?

What does the error mean and how can I fix it?

推荐答案

第一对建议:

1. 您应始终检查项目的维护情况.显然不是wget软件包.
2. 如果发生这种情况，您应该检查使用了哪个库.

1. You should always check how well is project maintained. Apparently wget package is not.
2. You should check which libs is package using, in case something like this happens.

现在，解决这个问题.

Now, to the issue.

显然，wget使用urllib.request进行请求.经过一些测试，我得出的结论是它不能处理所有HTTP状态代码.更具体地说，当HTTP状态为例如304时，它会以某种方式中断.这就是为什么必须使用具有更高级别接口的库的原因.甚至urllib.request也在官方文档中:

Apparently wget uses urllib.request for making request. After some testing, I concluded that it doesn't handle all HTTP status codes. More specifically, it somehow breaks when HTTP status is, for example, 304. This is why you have to use libraries with higher level interface. Even the urllib.request says this in official documentation:

对于更高级别的HTTP客户端界面，建议使用Requests程序包.

The Requests package is recommended for a higher-level HTTP client interface.

因此，事不宜迟，这是有效的代码段.

So, without further ado, here is the working snippet.

您只需更新要保存文件的位置即可.

You can just update with where you want to save files.

from multiprocessing import Pool

import shutil
import requests


def f(args):
    print(args)
    req = requests.get(args[1], stream=True)
    with open(str(args[0]), 'wb') as f:
        shutil.copyfileobj(req.raw, f)

if __name__ == "__main__":
    a = Pool(2)
    a.map(f, enumerate(urls))  # urls is a list of urls.

shutil lib用于文件操作.在这种情况下，将数据流传输到文件对象.

shutil lib is used for file manipulation. In this case, to stream the data to a file object.

这篇关于multiprocessing.pool.MaybeEncodingError:发送结果错误:原因:'TypeError(“无法序列化'_io.BufferedReader'object"，)'的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！