Python:通过使用线缓存与常规文件访问open()相比，读取大量文件 [英] Python: Huge file reading by using linecache Vs normal file access open()

问题描述

我处于一种情况，即多个线程使用多个指向同一文件的文件指针来读取相同的大文件.该文件将至少有100万行.每行的长度从500个字符到1500个字符不等.该文件将不会写入"操作.每个线程将从不同的行开始读取相同的文件.哪一种有效的方法..?使用Python的线缓存或普通的readline()还是还有其他有效的方法?

I am in a situation where multiple threads reading the same huge file with mutliple file pointers to same file. The file will have atleast 1 million lines. Eachline's length varies from 500 characters to 1500 characters. There won't "write" operations on the file. Each thread will start reading the same file from different lines. Which is the efficient way..? Using the Python's linecache or normal readline() or is there anyother effient way?

推荐答案

看看mmap模块: http://docs.python.org/library/mmap.html

它将允许您将文件用作数组，而OS则负责实际的读取和缓冲.

It will allow you to use the file as an array, while the OS handles the actual reading and buffering.

这篇关于Python:通过使用线缓存与常规文件访问open()相比，读取大量文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！