如何从用 7z 压缩的文本文件中读取? [英] How to read from a text file compressed with 7z?

问题描述

我想从 7z 压缩的 csv(文本)文件中逐行读取(在 Python 2.7 中).我不想解压缩整个(大)文件，而是要流式传输行.

I would like to read (in Python 2.7), line by line, from a csv (text) file, which is 7z compressed. I don't want to decompress the entire (large) file, but to stream the lines.

我尝试了 pylzma.decompressobj() 失败.我收到数据错误.请注意，此代码尚未逐行读取:

I tried pylzma.decompressobj() unsuccessfully. I get a data error. Note that this code doesn't yet read line by line:

input_filename = r"testing.csv.7z"
with open(input_filename, 'rb') as infile:
    obj = pylzma.decompressobj()
    o = open('decompressed.raw', 'wb')
    obj = pylzma.decompressobj()
    while True:
        tmp = infile.read(1)
        if not tmp: break
        o.write(obj.decompress(tmp))
    o.close()

输出:

    o.write(obj.decompress(tmp))
ValueError: data error during decompression

推荐答案

这将允许您迭代行.它部分源自我在另一个问题的答案中找到的一些代码.

This will allow you to iterate the lines. It's partially derived from some code I found in an answer to another question.

此时 (pylzma-0.5.0) py7zlib 模块没有实现允许存档成员作为数据流读取的 API字节或字符——它的 ArchiveFile 类只提供了一个 read() 函数，它一次解压并返回一个成员中未压缩的数据.鉴于此，最好的办法是通过 Python 生成器将其用作缓冲区，以迭代方式返回字节或行.

At this point in time (pylzma-0.5.0) the py7zlib module doesn't implement an API that would allow archive members to be read as a stream of bytes or characters — its ArchiveFile class only provides a read() function that decompresses and returns the uncompressed data in a member all at once. Given that, about the best that can be done is return bytes or lines iteratively via a Python generator using that as a buffer.

下面是后者，但如果问题是存档成员文件本身很大，则可能无济于事.

The following does the latter, but may not help if the problem is the archive member file itself is huge.

下面的代码应该可以在 Python 3.x 和 2.7 中使用.

The code below should work in Python 3.x as well as 2.7.

import io
import os
import py7zlib


class SevenZFileError(py7zlib.ArchiveError):
    pass

class SevenZFile(object):
    @classmethod
    def is_7zfile(cls, filepath):
        """ Determine if filepath points to a valid 7z archive. """
        is7z = False
        fp = None
        try:
            fp = open(filepath, 'rb')
            archive = py7zlib.Archive7z(fp)
            _ = len(archive.getnames())
            is7z = True
        finally:
            if fp: fp.close()
        return is7z

    def __init__(self, filepath):
        fp = open(filepath, 'rb')
        self.filepath = filepath
        self.archive = py7zlib.Archive7z(fp)

    def __contains__(self, name):
        return name in self.archive.getnames()

    def readlines(self, name, newline=''):
        r""" Iterator of lines from named archive member.

        `newline` controls how line endings are handled.

        It can be None, '', '
', '
', and '
' and works the same way as it does
        in StringIO. Note however that the default value is different and is to enable
        universal newlines mode, but line endings are returned untranslated.
        """
        archivefile = self.archive.getmember(name)
        if not archivefile:
            raise SevenZFileError('archive member %r not found in %r' %
                                  (name, self.filepath))

        # Decompress entire member and return its contents iteratively.
        data = archivefile.read().decode()
        for line in io.StringIO(data, newline=newline):
            yield line


if __name__ == '__main__':

    import csv

    if SevenZFile.is_7zfile('testing.csv.7z'):
        sevenZfile = SevenZFile('testing.csv.7z')

        if 'testing.csv' not in sevenZfile:
            print('testing.csv is not a member of testing.csv.7z')
        else:
            reader = csv.reader(sevenZfile.readlines('testing.csv'))
            for row in reader:
                print(', '.join(row))

这篇关于如何从用 7z 压缩的文本文件中读取?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！