如何在python中限制子进程stdout和stderr的大小 [英] How to limit the size of subprocess stdout and stderr in python

问题描述

我需要运行用户提交的应用程序.我的代码如下:

I need to run applications submitted by users. My code looks like:

def run_app(app_path):
    inp = open("app.in", "r")
    otp = open("app.out", "w")

    return subprocess.call(app_path, stdout=otp, stdin=inp)

现在，由于我无法控制用户提交的内容，因此我想限制应用程序输出的大小.试图访问未经授权的系统资源和滥用CPU周期等其他事情都受到保护者规则实施的限制.允许运行的最大时间由父进程处理(在python中).现在，流氓应用程序仍然可以通过将大量数据写入其标准输出来尝试淹没服务器系统，这是因为知道该标准输出已保存到文件中.

Now since I have no control over what users will submit, I want to restrict the size of the output of the application. Other things like trying to access unauthorized system resources and abusing of CPU cycles are being restricted by apparmor rule enforcement. The maximum time allowed to run is being handled by parent process (in python). Now a rogue application can still try to flood the server system by writing a lot of data to its stdout knowing the the stdout is being saved to a file.

我不想将AppArmors RLIMIT或内核模式下的任何东西用于stdout/stderr文件.能够使用标准库从python做到这一点非常好.

I do not want to use AppArmors RLIMIT or anything in kernel mode for stdout/stderr files. It would be great to be able to do it from python using standard library.

我目前正在考虑创建文件的子类，并在每次写入时检查已经向流中写入了多少数据.或创建具有最大长度设置的内存映射文件.

I am currently thinking about creating a subclass of file and on each write check how much data has been written already to the stream. Or create a memory mapped file with maximum length set.

但是我感觉可能有一种更简单的方法来限制文件大小，但我还没有看到.

But I am feeling there may be a simpler way to restrict file size I do not see it yet.

推荐答案

对file进行子类化或创建其他伪文件Python对象根本不起作用，因为该文件将在子进程中使用-因此必须一个操作系统文件，而不是Python类对象.子进程不会将您的Python对象发送给其他进程使用.

Subclassing file or creating other pseudo-file Python object will not work at all, since the file will be consumed in the subprocess - and therefore it must be an O.S. file, not a Python class object. Subprocess won't send your Python object for the other process to use.

虽然Python对内存映射文件提供了本机和简单的支持，但通过mmap模块，内存映射并不意味着:您可以指定镜像到内存的文件的大小，但是您可以这样做完全不写文件:多余的数据将被简单地写入磁盘，而不会被映射. (然后，再次将磁盘文件而不是mmap对象传递给子进程).可以在某个时候创建​​一个带有哨兵值的文件，并保持线程检查哨兵是否被覆盖，这时它可能会杀死子进程-但我怀疑这样做是否可靠.

And while Python has native and simple support to memory-mapping files, through the mmap module, memory mapping is not meant for this: you can specify the size of the file that is mirrored to memory, but you that does not limit writting to the file at all: excess data will simply be written to disk and not be mapped. (And, again, you pass to the subprocess the disk-file, not the mmap object). It would be possible to create a file with a sentinel value at some point, and keep a thread checking if the sentinel is overwritten, at which point, it could kill the subprocess - but I doubt that would be reliable.

然后，有一些磁盘活动监视工具，例如inotify:您可以使用 pyinotify 传递给您主进程上的处理程序，只要访问该文件，该处理程序就会被调用.缺点:文件写入"没有事件-只是文件访问"-我不确定文件的增量写入是否会触发任何可能的事件.而且，尽管如此，如果子进程将在单个系统调用中完成所有写操作，那么无论如何您都将被通知为时已晚.

Then, there are disk-activity monitoring tools, such as inotify: you could use pyinotify to a handler on your main process that would be called whenever the file is accessed. The downside: there is no event for 'file write' - just 'file acessed' - I am not sure if any of the possible events would be triggered by incremental writting of the file. And, nonetheless, if the child process would do all of its writting in a single system call, you'd just be notified too late anyway.

因此，我能想到的是:在人为限制的文件系统中创建文件.这样，当超过最大大小时，操作系统将阻止写入.

So, what I can think of that would work is: create a file in an artificially limited file-system. That way, the OS will block writing when max-size is exceeded.

在Linux下，您可以预创建具有所需大小+一些开销的文件，在其上创建FS，然后使用循环"接口将其挂载-然后只需在该文件系统内创建stdout和sterr文件即可，给您的孩子打电话.

Under Linux, you can pre-create a file with the desired size + some overhead, create an FS on it, and mount it with the "loop" interface - then just create your stdout and sterr files inside that filesystem, and call your child process.

您可以预先创建并预装载此类文件系统的池，以根据需要使用-甚至可以动态创建它们-但这需要以下步骤:创建FS主机文件，创建文件系统在其上进行结构化(mkfs)并进行安装-所有这些都可能会产生大量开销.

You could pre-create and pre-mount a pool of such filesystems to be used as needed - or, you could even create them dynamically - but that would require the steps of creating the FS host file, creating a file-system structure on it (mkfs) and mounting it - all of this could be a lot of overhead.

总而言之，也许您最好只使用Apparmor自己的rlimit设置.

All in all, maybe you are better simply using Apparmor's own rlimit settings.

这篇关于如何在python中限制子进程stdout和stderr的大小的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！