内存错误 - 在非常大的HTML文件，使用BeautifulSoup？ [英] Using BeautifulSoup on very large HTML file - memory error?

问题描述

我通过一个项目的工作学习Python的。我下载我的数据，其中包括我的所有消息的messages.htm文件。我想编写一个程序来分析这个文件和输出数据（消息＃，最常见的词，等等。）

I'm learning Python by working on a project - a Facebook message analyzer. I downloaded my data, which includes a messages.htm file of all my messages. I'm trying to write a program to parse this file and output data (# of messages, most common words, etc.)

不过，我messages.htm文件是270MB。当创建在外壳进行测试BeautifulSoup对象，任何其他文件（全部＆LT; 1MB）的作品就好了。但我不能创建messages.htm的学士学位对象。这里的错误：

However, my messages.htm file is 270MB. When creating a BeautifulSoup object in the shell for testing, any other file (all < 1MB) works just fine. But I can't create a bs object of messages.htm. Here's the error:

>>> mf = open('messages.htm', encoding="utf8")
>>> ms = bs4.BeautifulSoup(mf)
Traceback (most recent call last):
  File "<pyshell#73>", line 1, in <module>
    ms = bs4.BeautifulSoup(mf)
  File "C:\Program Files (x86)\Python\lib\site-packages\bs4\__init__.py", line 161, in __init__
markup = markup.read()
  File "C:\Program Files (x86)\Python\lib\codecs.py", line 319, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
MemoryError

所以我不能甚至开始使用此文件的工作。这是我第一次处理这样的事情，我才刚刚学习Python的，所以任何建议将是多少AP preciated！

So I can't even begin working with this file. This is my first time tackling something like this and I'm only just learning Python so any suggestions would be much appreciated!

推荐答案

当你正在使用这个作为一个学习的过程，我不会给太多code。你可能会与的ElementTree的iterparse ，让你处理，你解析。 BeautifulSoup没有这个功能就我所知。

As you're using this as a learning exercise, I won't give too much code. You may be better off with ElementTree's iterparse to allow you to process as you parse. BeautifulSoup doesn't have this functionality as far as I am aware.

为您开始：

import xml.etree.cElementTree as ET

with open('messages.htm') as source:

    # get an iterable
    context = ET.iterparse(source, events=("start", "end"))

    # turn it into an iterator
    context = iter(context)

    # get the root element
    event, root = context.next()

    for event, elem in context:
        # do something with elem

        # get rid of the elements after processing
        root.clear()

如果您正在使用BeautifulSoup设置，你可以看看分裂HTML源成可管理的块，但你必须要小心，以保持线程消息结构，并确保你保持有​​效的HTML。

If you're set on using BeautifulSoup, you could look into splitting the source HTML into manageable chunks, but you'd need to be careful to keep the thread-message structure and ensure you keep valid HTML.

这篇关于内存错误 - 在非常大的HTML文件，使用BeautifulSoup？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！