如何在AWS Glue上将消息写入输出日志? [英] How do I write messages to the output log on AWS Glue?
问题描述
AWS Glue作业默认将输出和错误记录到两个不同的CloudWatch日志/aws-glue/jobs/error
和/aws-glue/jobs/output
.当我在脚本中包含print()
语句进行调试时,它们将被写入错误日志(/aws-glue/jobs/error
).
AWS Glue jobs log output and errors to two different CloudWatch logs, /aws-glue/jobs/error
and /aws-glue/jobs/output
by default. When I include print()
statements in my scripts for debugging, they get written to the error log (/aws-glue/jobs/error
).
我尝试使用:
log4jLogger = sparkContext._jvm.org.apache.log4j
log = log4jLogger.LogManager.getLogger(__name__)
log.warn("Hello World!")
但是"Hello World!"没有显示在我运行的测试作业的任何日志中.
but "Hello World!" doesn't show up in either of the logs for the test job I ran.
有人知道如何将调试日志语句写入输出日志(/aws-glue/jobs/output
)吗?
Does anyone know how to go about writing debug log statements to the output log (/aws-glue/jobs/output
)?
TIA!
事实证明上面的方法确实有效.发生的事情是我在AWS Glue脚本编辑器窗口中运行该作业,该窗口捕获Command-F组合键,并且仅在当前脚本中进行搜索.因此,当我尝试在页面中搜索日志输出时,似乎好像没有对其进行日志记录.
It turns out the above actually does work. What was happening was that I was running the job in the AWS Glue Script editor window which captures Command-F key combinations and only searches in the current script. So when I tried to search within the page for the logging output it seemed as if it hadn't been logged.
注意:我确实通过测试第一响应者的建议发现,AWS Glue脚本似乎没有输出任何级别低于WARN的日志消息!
NOTE: I did discover through testing the first responder's suggestion that AWS Glue scripts don't seem to output any log message with a level less than WARN!
推荐答案
尝试使用logging
模块中的内置python记录器,默认情况下它将消息写入标准输出流.
Try to use built-in python logger from logging
module, by default it writes messages to standard output stream.
import logging
MSG_FORMAT = '%(asctime)s %(levelname)s %(name)s: %(message)s'
DATETIME_FORMAT = '%Y-%m-%d %H:%M:%S'
logging.basicConfig(format=MSG_FORMAT, datefmt=DATETIME_FORMAT)
logger = logging.getLogger(<logger-name-here>)
logger.setLevel(logging.INFO)
...
logger.info("Test log message")
这篇关于如何在AWS Glue上将消息写入输出日志?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!