我应该如何在AWS Lambda实现中对待joblib多处理? [英] How should I treat joblib multiprocessing in an AWS lambda implementation?

问题描述

我在AWS中有一个相对简单的线性回归lambda.每个实例中称为函数的日志显示以下内容:

I have a relatively simple linear regression lambda in AWS. Each instance the function is called the logs display the following:

/opt/python/sklearn/externals/joblib/_multiprocessing_helpers.py:38: UserWarning: [Errno 38] Function not implemented. joblib will operate in serial mode

warnings.warn('%s. joblib will operate in serial mode' % (e,))

我怀疑这是由于sklearn在lambda(即无服务器")上运行，并试图根据此GH问题.

I suspect this is due to sklearn running on a lambda (i.e. 'serverless') and trying to determine it's multi-processing capabilities as per this question and this GH issue.

我还从GH那里了解到这不是一个可修复"的问题，当在这些硬件上具有这些依赖项进行部署时，总是会发生这种情况.我恢复了预期的结果(即使我当前已将默认的最小lambda内存最大化为128mb).

I am also understanding from the GH that this is not a 'fixable' issue, it will always happen when deploying with these dependencies on this hardware. I am getting back my expected results (even though I am currently maxing out the default, minimum lambda memory of 128mb).

我的目标是控制警告，并且会知道是否有办法进行以下操作:

I aim to control the warnings and would know if there is a way to either:

• 停止sklearn寻找多处理程序，以防止发出警告
• 捕获此特定警告，并防止其从我的函数传递到cloudwatch日志中
• 如果两者都可行，那么从aws体系结构/pythonic观点来看这是更可取的吗?

• stop sklearn looking for multiprocessing, so preventing the warning from issuing
• capture this specific warning and prevent it from being passed from my function into the cloudwatch logs
• if both are possible, which would be preferable from a aws architecture/pythonic opinion?

推荐答案

要捕获警告并防止其传递到cloudwatch日志中，可以按以下方式过滤警告.

To capture the warning and prevent it from being passed into the cloudwatch logs, you can filter the warning as follows.

import json
import warnings
warnings.filterwarnings('error') 
try:
    import sklearn
except Warning:
    pass 

def lambda_handler(event, context):
    # TODO implement
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

此处的文章，尤其是最后，重新创建并过滤警告.

The article here, particularly towards the end, recreates and filters the warning.

这篇关于我应该如何在AWS Lambda实现中对待joblib多处理?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！