使用 subprocess.Popen 的非常大的输入和管道 [英] Very large input and piping using subprocess.Popen

问题描述

我有一个很简单的问题.我有一个大文件，它经过三个步骤，一个使用外部程序的解码步骤，在 python 中的一些处理，然后使用另一个外部程序重新编码.我一直在使用 subprocess.Popen() 尝试在 python 中执行此操作，而不是形成 unix 管道.但是，所有数据都缓冲到内存中.有没有一种 Python 的方式来完成这项任务，或者我最好退回到一个简单的 Python 脚本，该脚本从标准输入读取并在任一侧使用 unix 管道写入标准输出?

I have pretty simple problem. I have a large file that goes through three steps, a decoding step using an external program, some processing in python, and then recoding using another external program. I have been using subprocess.Popen() to try to do this in python rather than forming unix pipes. However, all the data are buffered to memory. Is there a pythonic way of doing this task, or am I best dropping back to a simple python script that reads from stdin and writes to stdout with unix pipes on either side?

import os, sys, subprocess

def main(infile,reflist):
    print infile,reflist
    samtoolsin = subprocess.Popen(["samtools","view",infile],
                                  stdout=subprocess.PIPE,bufsize=1)
    samtoolsout = subprocess.Popen(["samtools","import",reflist,"-",
                                    infile+".tmp"],stdin=subprocess.PIPE,bufsize=1)
    for line in samtoolsin.stdout.read():
        if(line.startswith("@")):
            samtoolsout.stdin.write(line)
        else:
            linesplit = line.split("	")
            if(linesplit[10]=="*"):
                linesplit[9]="*"
            samtoolsout.stdin.write("	".join(linesplit))

推荐答案

Popen 有一个 bufsize 参数，它将限制内存中缓冲区的大小.如果您根本不希望内存中的文件，您可以将文件对象作为 stdin 和 stdout 参数传递.来自 子进程文档:

Popen has a bufsize parameter that will limit the size of the buffer in memory. If you don't want the files in memory at all, you can pass file objects as the stdin and stdout parameters. From the subprocess docs:

bufsize，如果给定，与内置 open() 函数的相应参数具有相同的含义:0 表示无缓冲，1 表示行缓冲，任何其他正值表示使用(大约)该大小的缓冲区.负 bufsize 表示使用系统默认值，通常表示完全缓冲.bufsize 的默认值为 0(无缓冲).

bufsize, if given, has the same meaning as the corresponding argument to the built-in open() function: 0 means unbuffered, 1 means line buffered, any other positive value means use a buffer of (approximately) that size. A negative bufsize means to use the system default, which usually means fully buffered. The default value for bufsize is 0 (unbuffered).

这篇关于使用 subprocess.Popen 的非常大的输入和管道的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！