使用py4j在Java/python项目中将Log4j连接到ipython Notebook stderr [英] Connecting Log4j to ipython notebook stderr in a java/python project using py4j

问题描述

我有一个使用java，scala和Apache Spark进行基因组数据分布式计算的项目.使用py4j并模仿PySpark模型，我们公开了调用JVM的python API.我们的目标是将这种模型引入jupyter笔记本电脑，到目前为止，这非常容易，但存在一个挥之不去的问题:日志记录.

I have a project that uses java, scala and Apache Spark to do distributed computations on genomic data. Using py4j and mimicking the PySpark model, we expose a python API that calls into the JVM. Our goal has been to bring this model into jupyter notebooks, which has been pretty easy so far, with one lingering problem: logging.

我们(和Spark)使用log4j将日志消息写入日志文件和stderr.此stderr是Java进程的stderr，因此，如果我从jupyter笔记本中运行两个命令:

We (and Spark) use log4j to write log messages to a log file and stderr. This stderr is the stderr for the java process, so if I run two commands from the jupyter notebook:

print('foo')
info('bar')  # calls log4j logger.info in JVM

我看到'foo'写入了jupyter单元，但是'bar'被写入了运行jupyter进程的终端.

I see 'foo' written to the jupyter cell, but 'bar' is written to the terminal running the jupyter process.

将log4j连接到jupyter笔记本，以便将log4j消息写入jupyter单元，而不是终端.

Connect log4j to the jupyter notebook so that log4j messages are written to jupyter cells, instead of the terminal.

Java log4j.ConsoleAppender正在写入Java stderr.因此，我们将需要以某种方式将Java stderr路由通过jupyter，对吗?这可能涉及将System.setOut(...)与连接到jupyter进程的PrintStream对象一起使用，但是我不确定如何做到这一点.

The java log4j.ConsoleAppender is writing to the java stderr. So, we're going to need to route the java stderr through jupyter somehow, right? This may involve using System.setOut(...) with a PrintStream object hooked up to the jupyter process, but I'm not yet sure how to do that.

推荐答案

我们通过使用单独的套接字在Java和Python之间进行通信来解决了这一问题.这是提交差异: https://github.com/hail-is/hail/commit /93d7e95a82ab39501eede7ecb301538bcd013ea8

We solved this by using a separate socket to communicate between Java and Python. Here's the commit diff: https://github.com/hail-is/hail/commit/93d7e95a82ab39501eede7ecb301538bcd013ea8

这篇关于使用py4j在Java/python项目中将Log4j连接到ipython Notebook stderr的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！