超过最大递归深度.多处理和 bs4 [英] Maximum recursion depth exceeded. Multiprocessing and bs4

问题描述

我正在尝试使解析器使用 beautifulSoup 和多处理.我有一个错误:

I'm trying to make a parser use beautifulSoup and multiprocessing. I have an error:

RecursionError:超过最大递归深度

RecursionError: maximum recursion depth exceeded

我的代码是:

import bs4, requests, time
from multiprocessing.pool import Pool

html = requests.get('https://www.avito.ru/moskva/avtomobili/bmw/x6?sgtd=5&radius=0')
soup = bs4.BeautifulSoup(html.text, "html.parser")

divList = soup.find_all("div", {'class': 'item_table-header'})


def new_check():
    with Pool() as pool:
        pool.map(get_info, divList)

def get_info(each):
   pass

if __name__ == '__main__':
    new_check()

为什么会出现此错误以及如何解决?

Why I get this error and how I can fix it?

更新:错误的所有文本是

Traceback (most recent call last):
  File "C:/Users/eugen/PycharmProjects/avito/main.py", line 73, in <module> new_check()
  File "C:/Users/eugen/PycharmProjects/avito/main.py", line 67, in new_check
    pool.map(get_info, divList)
  File "C:UserseugenAppDataLocalProgramsPythonPython36libmultiprocessingpool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:UserseugenAppDataLocalProgramsPythonPython36libmultiprocessingpool.py", line 644, in get
    raise self._value
  File "C:UserseugenAppDataLocalProgramsPythonPython36libmultiprocessingpool.py", line 424, in _handle_tasks
    put(task)
  File "C:UserseugenAppDataLocalProgramsPythonPython36libmultiprocessingconnection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "C:UserseugenAppDataLocalProgramsPythonPython36libmultiprocessing
eduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
RecursionError: maximum recursion depth exceeded

推荐答案

当你使用 multiprocessing 时，你传递给 worker 的一切都必须是 腌制.

When you use multiprocessing, everything you pass to a worker has to be pickled.

不幸的是，许多BeautifulSoup 树无法腌制.

Unfortunately, many BeautifulSoup trees can't be pickled.

这有几个不同的原因.其中一些是已经修复的错误，因此您可以尝试确保您拥有最新的 bs4 版本，而一些特定于不同的解析器或树构建器......但很有可能没有这样的会有所帮助.

There are a few different reasons for this. Some of them are bugs that have since been fixed, so you could try making sure you have the latest bs4 version, and some are specific to different parsers or tree builders… but there's a good chance nothing like this will help.

但根本问题是树中的许多元素都包含对树其余部分的引用.

But the fundamental problem is that many elements in the tree contain references to the rest of the tree.

有时，这会导致实际的无限循环，因为循环引用对于其循环引用检测来说过于间接.但这通常是修复的错误.

Occasionally, this leads to an actual infinite loop, because the circular references are too indirect for its circular reference detection. But that's usually a bug that gets fixed.

但是，更重要的是，即使循环不是无限，它仍然可以从树的其余部分拖入 1000 多个元素，这已经足以导致递归错误.

But, even more importantly, even when the loop isn't infinite, it can still drag in more than 1000 elements from all over the rest of the tree, and that's already enough to cause a RecursionError.

我认为后者就是这里发生的事情.如果我使用你的代码并尝试pickle divList[0]，它会失败.(如果我提高递归限制并计算帧数，它需要 23080 的深度，这远远超过默认值 1000.)但是如果我采用完全相同的 div 并解析单独，它成功没有问题.

And I think the latter is what's happening here. If I take your code and try to pickle divList[0], it fails. (If I bump the recursion limit way up and count the frames, it needs a depth of 23080, which is way, way past the default of 1000.) But if I take that exact same div and parse it separately, it succeeds with no problem.

因此，一种可能性是仅执行 sys.setrecursionlimit(25000).这将解决这个确切页面的问题，但稍微不同的页面可能需要更多.(另外，将递归限制设置得这么高通常不是一个好主意——不是因为浪费了内存，而是因为它意味着实际的无限递归需要 25 倍的时间和 25 倍的浪费资源来检测.)

So, one possibility is to just do sys.setrecursionlimit(25000). That will solve the problem for this exact page, but a slightly different page might need even more than that. (Plus, it's usually not a great idea to set the recursion limit that high—not so much because of the wasted memory, but because it means actual infinite recursion takes 25x as long, and 25x as much wasted resources, to detect.)

另一个技巧是编写修剪树"的代码，在你pickle它之前/当你删除div的任何向上链接.这是一个很好的解决方案，只是它可能需要大量工作，并且需要深入了解 BeautifulSoup 的工作原理，我怀疑您是否愿意这样做.

Another trick is to write code that "prunes the tree", eliminating any upward links from the div before/as you pickle it. This is a great solution, except that it might be a lot of work, and requires diving into the internals of how BeautifulSoup works, which I doubt you want to do.

最简单的解决方法有点笨拙，但是……您可以将汤转换为字符串，将其传递给孩子，然后让孩子重新解析它:

The easiest workaround is a bit clunky, but… you can convert the soup to a string, pass that to the child, and have the child re-parse it:

def new_check():
    divTexts = [str(div) for div in divList]
    with Pool() as pool:
        pool.map(get_info, divTexts)

def get_info(each):
    div = BeautifulSoup(each, 'html.parser')

if __name__ == '__main__':
    new_check()

这样做的性能成本可能无关紧要；更大的担忧是，如果您的 HTML 不完美，转换为字符串并重新解析它可能不是一次完美的往返.因此，我建议您先在不进行多处理的情况下进行一些测试，以确保这不会影响结果.

The performance cost for doing this is probably not going to matter; the bigger worry is that if you had imperfect HTML, converting to a string and re-parsing it might not be a perfect round trip. So, I'd suggest that you do some tests without multiprocessing first to make sure this doesn't affect the results.

这篇关于超过最大递归深度.多处理和 bs4的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！