如何从我的 Python Spark 脚本登录 [英] How do I log from my Python Spark script

问题描述

我有一个使用 spark-submit 运行的 Python Spark 程序.我想在其中放入日志语句.

I have a Python Spark program which I run with spark-submit. I want to put logging statements in it.

logging.info("This is an informative message.")
logging.debug("This is a debug message.")

我想使用 Spark 使用的同一个记录器，以便日志消息以相同的格式输出，并且级别由相同的配置文件控制.我该怎么做?

I want to use the same logger that Spark is using so that the log messages come out in the same format and the level is controlled by the same configuration files. How do I do this?

我尝试将 logging 语句放在代码中，并从 logging.getLogger() 开始.在这两种情况下，我都看到了 Spark 的日志消息，但没有看到我的.我一直在查看 Python 日志记录文档，但一直无法从

I've tried putting the logging statements in the code and starting out with a logging.getLogger(). In both cases I see Spark's log messages but not mine. I've been looking at the Python logging documentation, but haven't been able to figure it out from there.

不确定这是提交给 Spark 的脚本所特有的，还是只是我不了解日志记录的工作原理.

Not sure if this is something specific to scripts submitted to Spark or just me not understanding how logging works.

推荐答案

您可以从 SparkContext 对象中获取记录器:

You can get the logger from the SparkContext object:

log4jLogger = sc._jvm.org.apache.log4j
LOGGER = log4jLogger.LogManager.getLogger(__name__)
LOGGER.info("pyspark script logger initialized")

这篇关于如何从我的 Python Spark 脚本登录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！