在Linux中使用python分配特定大小的文件 [英] Allocate a file of particular size in Linux with python

问题描述

我正在用python编写I/O密集型程序，我需要在硬盘上分配特定数量的存储.由于我需要尽可能快，所以我不想在循环中创建内容为零(或虚拟)的文件. python是否有任何库或方法可以这样做，或者我必须在python中使用Linux命令?

I am writing an I/O intensive program in python and I need to allocate a specific amount of storage on hard disk. Since I need to be as fast as possible I do not want to make a file with zero (or dummy) content in a loop. Does python have any library or method to do so, or do I have to use a Linux command in python?

实际上，我正在实现一个类似于BitTorrent的应用程序.在我的代码中，接收方将源文件的每个段存储在一个单独的文件中(源文件的每个段都来自一个随机发送方).最后，所有单独的文件将被合并.这样做需要很多时间.

Actually, I am implementing an application that works like BitTorrent. In my code, the receiver stores every segment of the source file in a separate file (each segment of the source file comes from a random sender). At the end, all the separate files will be merged. It takes lots of time to do so.

因此，我想预先分配一个文件，然后将接收到的源文件的每个段写入其在预分配文件中的偏移量中.

Therefore, I want to allocate a file in advance and then write every received segment of the source file in its offset in the pre-allocated file.

def handler(self):    
    BUFFER_SIZE = 1024  # Normally 1024, but we want fast response
    # self.request is the TCP socket connected to the client
    data = self.request.recv(BUFFER_SIZE)
    addr = ..... #Some address

    details = str(data).split() 
    currentFileNum = int(details[0]) #Specifies the segment number of the received file.
    totalFileNumber = int(details[1].rstrip('\0')) # Specifies the total number of the segments that should be received.
    print '\tReceive: Connection address:', addr,'Current segment Number: ', currentFileNum, 'Total Number of file segments: ', totalFileNumber

    f = open(ServerThreadHandler.fileOutputPrefix + '_Received.%s' % currentFileNum, 'wb')
    data = self.request.recv(BUFFER_SIZE)
    while (data and data != 'EOF'):
        f.write(data)
        data = self.request.recv(BUFFER_SIZE)
    f.close()
    print "Done Receiving." ," File Number: ", currentFileNum
    self.request.sendall('\tThank you for data. File Number: ' + str(currentFileNum))
    ServerThreadHandler.counterLock.acquire()
    ServerThreadHandler.receivedFileCounter += 1
    if ServerThreadHandler.receivedFileCounter == totalFileNumber:
        infiles = []
        for i in range(0, totalFileNumber):
            infiles.append(ServerThreadHandler.fileOutputPrefix + '_Received.%s' % i)

        File_manipulation.cat_files(infiles, ServerThreadHandler.fileOutputPrefix + ServerThreadHandler.fileOutputSuffix, BUFFER_SIZE) # It concatenates the files based on their segment numbers. 
    ServerThreadHandler.counterLock.release()

推荐答案

通常(不仅在Python中，而且在OS级别上)现代FS驱动程序都支持

Generally (not only in Python but on the OS level) modern FS drivers support sparse files when you pre-create an apparently zero-filled file and then perform seek-and-write cycles to a point where you need to write a particular bit of data.

请参见如何创建带有文件孔的文件?了解如何创建此类文件.

See How to create a file with file holes? to understand how to create such a file.

这篇关于在Linux中使用python分配特定大小的文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！