使用子过程捕获Jupyter-Notebook标准输出 [英] capture jupyter-notebook stdout with subprocess

问题描述

我正在创建一个工具，允许用户在AWS服务器上运行带有pyspark的jupyter-notebook，并将端口转发到其本地主机以连接到笔记本.

I'm working on creating a tool that allows users to run a jupyter-notebook w/ pyspark on an AWS server and forward the port to their localhost to connect to the notebook.

我一直在使用subprocess.Popen ssh进入远程服务器并启动pyspark shell/笔记本，但是我无法避免让它将所有内容打印到终端.我想每行执行一个操作以检索端口号.

I've been using subprocess.Popen to ssh into the remote server and kick off the pyspark shell/notebook, but I'm unable to avoid having it print everything to the terminal. I WANT to perform an action per line to retrieve the port number.

例如，运行此代码(遵循此处最受欢迎的答案:从subprocess.communicate())

For example, running this (following the most popular answer here: Read streaming input from subprocess.communicate())

command = "jupyter-notebook"
con = subprocess.Popen(['ssh', node, command], stdout=subprocess.PIPE, bufsize=1)

with con.stdout:
    for line in iter(con.stdout.readline, b''):
        print(line),
con.wait()

这将忽略上下文管理器，并且con部分开始从stdout打印，以便立即将其打印到终端

this ignores the context manager, and the con portion starts printing off stdout so that this is immediately printed to terminal

[I 16:13:20.783 NotebookApp] [nb_conda_kernels] enabled, 0 kernels found
[I 16:13:21.031 NotebookApp] JupyterLab extension loaded from /home/*****/miniconda3/envs/aws/lib/python3.7/site-packages/jupyterlab
[I 16:13:21.031 NotebookApp] JupyterLab application directory is /data/data0/home/*****/miniconda3/envs/aws/share/jupyter/lab
[I 16:13:21.035 NotebookApp] [nb_conda] enabled
...
...
...

当我调用如下所示的随机脚本而不是"jupyter-notebook"(其中command="bash random_script.sh")时，我可以使上下文管理器发挥作用

I can get the context manager to function when I call a random script like the below instead of "jupyter-notebook" (where command="bash random_script.sh")

# random_script.sh
for i in $(seq 1 100)
do
    echo "some output: $i"
    sleep 2
done

这按预期方式运行，实际上我可以在with语句中每行执行一个操作. jupyter版本在本质上有什么不同之处，可防止其行为类似?

This acts as expected, and I can actually perform an action per line within the with statement. Is there something fundamentally different about the jupyter version that prevents this from acting similarly?

推荐答案

事实证明，该问题与jupyter产生的控制台输出实际上将输出到STDERR而不是stdout有关.我不知道为什么.但是无论如何，此更改完全解决了该问题:

The issue turned out to have everything to do with the fact that the console output produced by jupyter was actually going to STDERR instead of stdout. I'm not sure why. But regardless, this change totally fixed the issue:

con = subprocess.Popen(['ssh', node, command], 
                       stdout=subprocess.PIPE, 
                       stderr=subprocess.STDOUT,  # <-- redirect stderr to stdout
                       bufsize=1)

这篇关于使用子过程捕获Jupyter-Notebook标准输出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！