使用 Python 子进程处理交互式 shell [英] Handling interactive shells with Python subprocess

问题描述

我正在尝试使用多处理池来运行基于控制台的游戏的多个实例(地牢爬行石汤——自然用于研究目的)来评估每次运行.

I am trying to run multiple instances of a console-based game (dungeon crawl stone soup -- for research purposes naturally) using a multiprocessing pool to evaluate each run.

过去，当我使用池来评估相似的代码(遗传算法)时，我使用 subprocess.call 来拆分每个进程.但是，由于 dcss 具有很强的交互性，因此共享子 shell 似乎存在问题.

In the past when I've used a pool to evaluate similar code (genetic algorithms), I've used subprocess.call to split off each process. However, with dcss being quite interactive having a shared subshell seems to be problematic.

我有我通常用于这种事情的代码，爬行取代了我抛出 GA 的其他应用程序.有没有比这更好的方法来处理高度交互的 shell?我曾考虑为每个实例启动一个屏幕，但认为有一种更清洁的方法.我的理解是 shell=True 应该产生一个子外壳，但我想我以每次调用之间共享的方式产生一个.

I have the code I normally use for this kind of thing, with crawl replacing other applications I've thrown a GA at. Is there a better way to handle highly-interactive shells than this? I'd considered kicking off a screen for each instance, but thought there was a cleaner way. My understanding was that shell=True should be spawning a sub-shell, but I guess I it is spawning one in a way that is shared between each call.

我应该提到我有一个运行游戏的机器人，所以我不希望用户端发生任何实际交互.

I should mention I have a bot running the game, so I don't want any actual interaction from the user's end to occur.

# Kick off the GA execution
pool_args = zip(trial_ids,run_types,self.__population)
pool.map(self._GAExecute, pool_args)

---

# called by pool.map 
def _GAExecute(self,pool_args):
  trial_id       = pool_args[0]
  run_type       = pool_args[1]
  genome         = pool_args[2]
  self._RunSimulation(trial_id)

# Call the actual binary
def _RunSimulation(self, trial_id):
  command = "./%s" % self.__crawl_binary
  name    = "-name %s" % trial_id
  rc      = "-rc %s" % os.path.join(self.__output_dir,'qw-%s'%trial_id,"qw -%s.rc"%trial_id)
  seed    = "-seed %d" % self.__seed
  cdir    = "-dir %s" % os.path.join(self.__output_dir,'qw-%s'%trial_id)

  shell_command = "%s %s %s %s %s" % (command,name,rc,seed,cdir)
  call(shell_command, shell=True)

推荐答案

您确实可以将 stdin 和 stdout 关联到文件，如@napuzba 的回答所示:

You can indeed associate stdin and stdout to files, as in the answer from @napuzba:

fout = open('stdout.txt','w')
ferr = open('stderr.txt','w')
subprocess.call(cmd, stdout=fout, stderr=ferr)

另一种选择是使用 Popen 而不是调用.不同之处在于调用等待完成(阻塞)而 Popen 不是，参见 子进程 Popen 和 call 有什么区别(我如何使用它们)?

Another option would be to use Popen instead of call. The difference is that call waits for completion (is blocking) while Popen is not, see What's the difference between subprocess Popen and call (how can I use them)?

使用 Popen，您可以将 stdout 和 stderr 保留在您的对象中，然后在以后使用它们，而无需依赖文件:

Using Popen, you can then keep stdout and stderr inside your object, and then use them later, without having to rely on a file:

p = subprocess.Popen(cmd,stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.wait()
stderr = p.stderr.read()
stdout = p.stdout.read()

此方法的另一个潜在优点是您可以运行多个 Popen 实例而无需等待完成而不是拥有线程池:

Another potential advantage of this method is that you could run multiple instances of Popen without waiting for completion instead of having a thread pool:

processes=[
  subprocess.Popen(cmd1,stdout=subprocess.PIPE, stderr=subprocess.PIPE),
  subprocess.Popen(cmd2,stdout=subprocess.PIPE, stderr=subprocess.PIPE),
  subprocess.Popen(cmd3,stdout=subprocess.PIPE, stderr=subprocess.PIPE)
]

for p in processes:
  if p.poll():
     # process completed
  else:
     # no completion yet

附带说明，您应该避免 shell=True 如果可以，如果你不使用它，Popen 期望一个列表作为一个命令而不是一个字符串.不要手动生成此列表，而是使用 shlex 它将处理所有为您准备的边角案例，例如:

On a side note, you should avoid shell=True if you can, and if you do not use it Popen expects a list as a command instead of a string. Do not generate this list manually, but use shlex which will take care of all corner cases for you, eg.:

Popen(shlex.split(cmd), stdout=subprocess.PIPE, stderr=subprocess.PIPE)

这篇关于使用 Python 子进程处理交互式 shell的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！