与C程序通信时，子进程Popen无效的参数/管道中断 [英] Subprocess Popen invalid argument/broken pipe while communicating to C program
问题描述
我有这个代码
所有必需的库均已导入
class VERTEX(Structure):
_fields_ = [("index", c_int),
("x", c_float),
("y", c_float)]
其他东西
这是从顶点列表中创建和排列的
def writelist_buf(size, nomeID):
Nvert_VERTEX_Array_Type = VERTEX * len(bpy.data.objects[nomeID].data.vertices)
passarr = Nvert_VERTEX_Array_Type()
for i in range(len(passarr)):
vert = bpy.data.objects[nomeID].data.vertices[i]
passarr[i] = VERTEX(vert.index, vert.co[0], vert.co[1])
return passarr
bpy.data.objects [nomeID] .data.vertices是一个顶点列表.
其他东西
这是在def内部，并与C程序通信以前的数组
input = writelist_buf(size, nomeID)
c_program_and_args = "here is the program with his arguments(it works)"
cproc = Popen(c_program_and_args, stdin=PIPE, stdout=PIPE)
out, err = cproc.communicate(input)
#the program returns 2 integers separed by a space
return [int(i) for i in out.decode.split()]
size和nomeID在调用writelist之前声明.
经过一些调试"后，我发现writelist_buf传递的类型是合法"(它是字节，因为是使用c_types创建的数组)，但是我一直收到Errno32断管或Errno22无效参数. .C程序只需读入stdiin即可检索所有顶点(如下面的C代码).
奇怪的想法是，在集成"我正在处理的代码之前，我尝试了一个更简单的代码:这个，它可以工作！
from subprocess import Popen, PIPE
from ctypes import *
class VERTEX(Structure):
_fields_ = [("index", c_int),
("x", c_float),
("y", c_float)]
nverts = 5
vlist = [VERTEX(0,1,1), VERTEX(1,2,2), VERTEX(2,3,3), VERTEX(3,4,4), VERTEX(4,5,5)]
array = VERTEX * nverts
input = array()
for i in range(nverts):
input[i] = vlist[i]
print(type(input))
cproc = Popen("pipeinout.exe random arg", stdin=PIPE, stdout=PIPE)
out, err = cproc.communicate(input)
print(out.decode())
还有C代码
#include<stdio.h>
#include<stdlib.h>
typedef struct {
int index;
float x;
float y;
} vertex;
int main(int argc, char* argv[]) {
int n=5;
int i;
printf("%s",argv[1]);
vertex* VV;
VV=(vertex*)malloc(sizeof(vertex)*n);
fread(VV,sizeof(vertex),n,stdin);
//fread(&VV,sizeof(VV),1,stdin);//metti nel valore di VV(non a quello che punta) l'indirizzo passato||sizeof(VV) is the size of a pointer
for(i=0;i<n;i++)
printf(" %i , %f , %f\n",VV[i].index,VV[i].x,VV[i].y);
}
从您的评论中我了解到，您数百次将数百万个项目传递给C程序.下面的方法(使用子过程进行管道输入)可能在您的情况下太慢了.可能的替代方法是编写C扩展名(例如，使用Cython)或使用ctypes直接调用C函数.您可以问一个单独的问题，详细描述您的用例，以了解哪种方法更可取.
如果您选择了一种方法，那么请确保在进行任何优化之前，该方法可以正常工作(编写一些测试，评估性能，并且仅在必要时对其进行优化之后)-源):
import struct, sys
from subprocess import Popen, PIPE
vertex_struct = struct.Struct('i f f')
def pack(vertices, n):
yield struct.pack('i', n)
for v in vertices:
yield vertex_struct.pack(*v)
def main():
try: n = int(sys.argv[1])
except IndexError:
n = 100
vertices = ((i,i+1,i+2) for i in range(n))
p = Popen(["./echo_vertices", "random", "arg"], stdin=PIPE, stdout=PIPE)
out, _ = p.communicate(b''.join(pack(vertices, n)))
index, x, y = vertex_struct.unpack(out)
assert index == (n-1) and int(x) == n and int(y) == (n+1)
if __name__ == '__main__':
main()
此处是对问题的评论中的代码.对于我的计算机上的大n值，它可以正常工作:
import struct, sys
from subprocess import Popen, PIPE
from threading import Thread
def pack(vertices, n):
yield struct.pack('i', n)
s = struct.Struct('i f f')
for v in vertices:
yield s.pack(*v)
def write(output_file, chunks):
for chunk in chunks:
output_file.write(chunk)
output_file.close()
def main():
try: n = int(sys.argv[1])
except IndexError:
n = 100
vertices = ((i,i+1,i+2) for i in range(n))
p = Popen(["./echo_vertices", "random", "arg"], stdin=PIPE, stdout=PIPE)
Thread(target=write, args=[p.stdin, pack(vertices, n)]).start()
for line in iter(p.stdout.readline, b''):
pass
p.stdout.close()
sys.stdout.buffer.write(line)
p.wait()
if __name__ == '__main__':
main()
问题与解答
问::我不太了解这个包 函数(我知道收益回报 可迭代的对象 只有一次，但是在您的代码中，您 用2的产量，所以我不知道它是什么 返回.
pack()是生成器.生成器对您的描述不起作用，例如:
>>> def f():
... yield 1
... yield 2
...
>>> for i in f():
... print(i)
...
1
2
请注意，每个yield都会产生一个值.
>>> def g(n):
... for i in range(n):
... yield i
...
>>> list(g(10))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
这是yield在文本中仅出现一次，但被执行10次，每次产生一个值(在这种情况下为整数).请参阅Python教程中的生成器. 针对系统程序员的生成器技巧" 包含多个示例，说明了如何使用从简单用法到高级用法的生成器.
问:另外，我不知道是什么 (* v)表示在第10行)
s.pack(*v)使用参数解压缩来调用pack方法:
>>> def h(a, b):
... print(a, b)
...
>>> h(*[1, 'a'])
1 a
>>> h(*range(2))
0 1
>>> h(0, 1)
0 1
问:我不知道如何 第25行中的线程起作用，
Thread(target=write, args=[p.stdin, pack(vertices, n)]).start()
此行启动一个新线程，该线程使用args关键字参数的参数(即output_file=p.stdin和chunks=pack(vertices, n))调用write()函数.在这种情况下，write()函数等效于:
p.stdin.write(struct.pack('i', n))
p.stdin.write(s.pack(0, 1, 2))
p.stdin.write(s.pack(1, 2, 3))
...
p.stdin.write(s.pack(n-1, n, n+1))
p.stdin.close()
此后线程退出.
问: ...以及所有 程序的读取输出.它 不是存储在变量中吗?
整个输出未存储在任何地方.代码:
for line in iter(p.stdout.readline, b''):
pass
逐行读取p.stdout，直到.readline()返回空字符串b''并将当前行存储在line变量中(请参见 http://docs.python.org/library/subprocess.html? #subprocess.Popen.wait
对于提供的C代码，没有必要在单独的线程中写入p.stdin.我完全使用线程是为了避免警告中描述的情况，即C程序在脚本完成向其stdin的写入之前会生成足够的输出(您的C代码在完成读取之前不会编写任何内容，因此该线程是不必要的). /p>
换句话说，在这种情况下p.wait()是安全的.
没有p.wait()的C程序的stderr输出可能会丢失.尽管我只能使用脚本在jython上重现stderr损失.再一次，对于所提供的C代码，这没关系，因为它没有向stderr写入任何内容.
I have this code
All the needed libraries are imported
class VERTEX(Structure):
_fields_ = [("index", c_int),
("x", c_float),
("y", c_float)]
Other stuff
This create and array from a list of vertex
def writelist_buf(size, nomeID):
Nvert_VERTEX_Array_Type = VERTEX * len(bpy.data.objects[nomeID].data.vertices)
passarr = Nvert_VERTEX_Array_Type()
for i in range(len(passarr)):
vert = bpy.data.objects[nomeID].data.vertices[i]
passarr[i] = VERTEX(vert.index, vert.co[0], vert.co[1])
return passarr
bpy.data.objects[nomeID].data.vertices is a list of vertices.
Other stuff
This is inside a def, and communicate to a C program the previous array
input = writelist_buf(size, nomeID)
c_program_and_args = "here is the program with his arguments(it works)"
cproc = Popen(c_program_and_args, stdin=PIPE, stdout=PIPE)
out, err = cproc.communicate(input)
#the program returns 2 integers separed by a space
return [int(i) for i in out.decode.split()]
size and nomeID are declared before the writelist call.
After a bit of "debugging" i found that the type passed by the writelist_buf is "legal"(it's bytes, since is an array created with c_types), but i keep receiving a Errno32 Broken Pipe or Errno22 Invalid argument... The C program just make a read in the stdiin to retrive all the vertices(like the C code below)..
The strange think is that before "integrating" inside the code i was working on, i have tried a simpler code: this one, and it works!
from subprocess import Popen, PIPE
from ctypes import *
class VERTEX(Structure):
_fields_ = [("index", c_int),
("x", c_float),
("y", c_float)]
nverts = 5
vlist = [VERTEX(0,1,1), VERTEX(1,2,2), VERTEX(2,3,3), VERTEX(3,4,4), VERTEX(4,5,5)]
array = VERTEX * nverts
input = array()
for i in range(nverts):
input[i] = vlist[i]
print(type(input))
cproc = Popen("pipeinout.exe random arg", stdin=PIPE, stdout=PIPE)
out, err = cproc.communicate(input)
print(out.decode())
And the C code
#include<stdio.h>
#include<stdlib.h>
typedef struct {
int index;
float x;
float y;
} vertex;
int main(int argc, char* argv[]) {
int n=5;
int i;
printf("%s",argv[1]);
vertex* VV;
VV=(vertex*)malloc(sizeof(vertex)*n);
fread(VV,sizeof(vertex),n,stdin);
//fread(&VV,sizeof(VV),1,stdin);//metti nel valore di VV(non a quello che punta) l'indirizzo passato||sizeof(VV) is the size of a pointer
for(i=0;i<n;i++)
printf(" %i , %f , %f\n",VV[i].index,VV[i].x,VV[i].y);
}
From your comments I understand that you pass millions of items hundreds of times to a C program. The approach below (pipe input using subprocess) might be too slow in your case. Possible alternatives could be to write a C extension (e.g., using Cython) or to use ctypes to call C functions directly. You could ask a separate question describing your use case in detail about what approach could be preferable.
If you've chosen an approach then make sure that it works correctly before any optimization (write some tests, measure performance and only after optimize it if needed) -- Make it work, make it right, make it fast.
On the other hand there is no point to invest too much time in approaches that are known to be thrown away later -- Fail fast.
if the output of the C program is bounded; the .communicate() method from your code works (source):
import struct, sys
from subprocess import Popen, PIPE
vertex_struct = struct.Struct('i f f')
def pack(vertices, n):
yield struct.pack('i', n)
for v in vertices:
yield vertex_struct.pack(*v)
def main():
try: n = int(sys.argv[1])
except IndexError:
n = 100
vertices = ((i,i+1,i+2) for i in range(n))
p = Popen(["./echo_vertices", "random", "arg"], stdin=PIPE, stdout=PIPE)
out, _ = p.communicate(b''.join(pack(vertices, n)))
index, x, y = vertex_struct.unpack(out)
assert index == (n-1) and int(x) == n and int(y) == (n+1)
if __name__ == '__main__':
main()
Here's the code from the comments to the question. It works without errors for large n values on my machine:
import struct, sys
from subprocess import Popen, PIPE
from threading import Thread
def pack(vertices, n):
yield struct.pack('i', n)
s = struct.Struct('i f f')
for v in vertices:
yield s.pack(*v)
def write(output_file, chunks):
for chunk in chunks:
output_file.write(chunk)
output_file.close()
def main():
try: n = int(sys.argv[1])
except IndexError:
n = 100
vertices = ((i,i+1,i+2) for i in range(n))
p = Popen(["./echo_vertices", "random", "arg"], stdin=PIPE, stdout=PIPE)
Thread(target=write, args=[p.stdin, pack(vertices, n)]).start()
for line in iter(p.stdout.readline, b''):
pass
p.stdout.close()
sys.stdout.buffer.write(line)
p.wait()
if __name__ == '__main__':
main()
Q&A
Q: I don't really understand the pack functions (i know that yield returns an iterable object that is iterable only one time, but in your code you use 2 yield, so i don't get what it returns.
pack() is a generator. Generators do not work how you've described them, e.g.:
>>> def f():
... yield 1
... yield 2
...
>>> for i in f():
... print(i)
...
1
2
Note each yield produces a value.
>>> def g(n):
... for i in range(n):
... yield i
...
>>> list(g(10))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Here's the yield is present in the text only one time but it is executed 10 times and each time it produces a value (an integer in this case). See Generators in the Python tutorial. "Generator Tricks for Systems Programmers" contains multiple examples on how to use generators from a simple to an advanced usage.
Q: In addition i dont know what (*v) means at line 10)
s.pack(*v) calls the pack method using argument unpacking:
>>> def h(a, b):
... print(a, b)
...
>>> h(*[1, 'a'])
1 a
>>> h(*range(2))
0 1
>>> h(0, 1)
0 1
Q: I don't get how the Thread in line 25 works,
Thread(target=write, args=[p.stdin, pack(vertices, n)]).start()
This line starts a new thread that calls write() function with the arguments from the args keyword argument i.e. output_file=p.stdin and chunks=pack(vertices, n). The write() function in this case is equivalent to:
p.stdin.write(struct.pack('i', n))
p.stdin.write(s.pack(0, 1, 2))
p.stdin.write(s.pack(1, 2, 3))
...
p.stdin.write(s.pack(n-1, n, n+1))
p.stdin.close()
After that the thread exits.
Q: ...and all the read output of the program.. It isn't stored in a variable, is it?
The whole output is not stored anywhere. The code:
for line in iter(p.stdout.readline, b''):
pass
reads from p.stdout line-by-line until the .readline() returns empty string b'' and stores the current line in the line variable (see iter() docs). So:
sys.stdout.buffer.write(line)
just prints the last line of the output.
Q: 1)after starting the Thread, the python script waits until it finished, right?
No, the main thread exits. The started thread is not daemon. It runs until it completes i.e., the script (the program) doesn't exit until it completes.
Q: 2)i understood how you read from the stdout of the C program,but i don't get when you start it.Afa i understood,with the write function we write in a buffer(or something like a file in the ram) the data we want,and when we run the c program, it can read from it the data we wrote.But when we start the C program in your code? :)
The C program is started by p = Popen(...).
p.stdin.write() writes to stdin of the C program (there are number of buffers in between but we can forget about it for a moment). The process is the same as in:
$ echo abc | some_program
Q: 3)last thing: why do you use a wait on p? There's a warning http://docs.python.org/library/subprocess.html?#subprocess.Popen.wait
For the provided C code it is not necessary to write to p.stdin in a separate thread. I use the thread exactly to avoid the situation described in the warning i.e., C program produces enough output before the script finishes writing to its stdin (your C code doesn't write anything before it finishes reading so the thread is not necessary).
In other words p.wait() is safe in this case.
Without p.wait() stderr output from the C program might be lost. Though I can reproduce the stderr loss only on jython with the scripts. Yet again for the provided C code it doesn't matter due to it is not writing to stderr anything.
这篇关于与C程序通信时，子进程Popen无效的参数/管道中断的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
