如何在Python Shell Glue作业中使用CloudWatch自定义日志组? [英] How to use a CloudWatch custom log group with Python Shell Glue job?

问题描述

我有一些"Python Shell"类型的Glue作业，我想将作业日志发送到自定义CloudWatch日志组，而不是默认日志组.我可以通过提供以下作业参数来实现火花"型胶水作业:

I have some "Python Shell" type Glue jobs and I want to send the job logs to a custom CloudWatch log group instead of the default log group. I am able to achieve this for "Spark" type glue jobs by providing job parameters as below:

"--enable-continuous-cloudwatch-log" = true
"--continuous-log-logGroup" = "/aws-glue/jobs/glue-job-1"

，但是相同的参数不适用于Python Shell作业(日志仍进入默认日志组/aws-glue/python-jobs/output和/aws-glue/python-jobs/error).有什么方法可以实现Python Shell作业吗?

but the same parameters doesn't work for Python Shell jobs (logs still going to the default log groups /aws-glue/python-jobs/output and /aws-glue/python-jobs/error). Is there any way to achieve this for Python Shell jobs?

推荐答案

continuous-log-logGroup 是AWS Glue Spark作业随附的，不适用于Python Shell作业.您可以做的最接近的事情是配置一个写入CloudWatch的日志处理程序. Watch望塔是一个受欢迎的炮台:

continuous-log-logGroup is something that comes with AWS Glue Spark jobs and it's not available to Python Shell jobs. The closest thing you can do is to configure a log handler that writes to CloudWatch. Watchtower is a popular one:

import watchtower, logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logger.addHandler(watchtower.CloudWatchLogHandler(log_group='watchtower', stream_name='bla'))
logger.info("Hi")
logger.info(dict(foo="bar", details={}))

您还可以直接使用Cloudwatch Logs API:

You can also directly use Cloudwatch Logs API:

logs = boto3.client('logs')

LOG_GROUP='TUTORIAL-DEV2'
LOG_STREAM='stream1'

logs.create_log_group(logGroupName=LOG_GROUP)
logs.create_log_stream(logGroupName=LOG_GROUP, logStreamName=LOG_STREAM)


timestamp = int(round(time.time() * 1000))

response = logs.put_log_events(
    logGroupName=LOG_GROUP,
    logStreamName=LOG_STREAM,
    logEvents=[
        {
            'timestamp': timestamp,
            'message': time.strftime('%Y-%m-%d %H:%M:%S')+'\tHello world, here is our first log message!'
        }
    ]
)

此示例来自以下要点: https://gist.github.com/olegdulin/fd18906343d75142a487b9a9da9042e0

This example was from this gist: https://gist.github.com/olegdulin/fd18906343d75142a487b9a9da9042e0

这篇关于如何在Python Shell Glue作业中使用CloudWatch自定义日志组?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！