Python中的AWS Lambda和Open CV的无服务器问题 [英] Serverless issue with AWS Lambda and Open CV in Python

问题描述

我正在开发一种微服务，以分析上传到S3 AWS Bucket的图像.我正在使用无服务器框架.我正在使用virtualenv通过PIP和serverless-python-requirements插件安装依赖项，以将这些依赖项部署到Lambda函数.

I am developing a microservice to analyze an image uploaded to an S3 AWS Bucket. I am using Serverless framework. I am using virtualenv to install the dependencies with PIP and serverless-python-requirements plugin to deploy these dependencies to the Lambda function.

但是，由于缺少.so文件，部署微服务时出现错误.我得到的错误是

However I am having an error when I deploy the microservice because of a missing .so file. The error I get is

无法导入模块'handlers.image':libgthread-2.0.so.0:无法打开共享对象文件:没有此类文件

我的serverless.yml文件是

My serverless.yml file is

service: sls-covid

provider:
  name: aws
  runtime: python3.8
  profile: testuser
  stage: ${opt:stage, 'staging'}
  environment: ${file(environment.yml):${self:provider.stage}}
  region: ${self:provider.environment.REGION}

  iamRoleStatements:
    # para poder leer y escribir en el bucket
    - Effect: "Allow"
      Action:
        - "s3:*"
      Resource: "*"

custom:
  pythonRequirements:
    dockerizePip: non-linux


package:
  individually: true
  exclude:
    - node_modules/**
    - env/**
    - package.json
    - package-lock.json

functions:

  analyzeImage:
    # para analizar la imagen cuando se almacena en S3
    handler: handlers.image.analyze
    events:
      - s3:
          bucket: ${self:service}-${self:provider.stage}-images
          event: s3:ObjectCreated:*
          rules:
            - prefix: uploads/

plugins:
  - serverless-python-requirements

相关代码为:

import json
import logging
import boto3
from pydicom import dcmread
import numpy as np
# here the code stops working
import cv2
from pydicom.filebase import DicomBytesIO

logger = logging.getLogger()
logger.setLevel(logging.INFO)

s3 = boto3.client('s3')


def analyze(event, context):

    print("=== event: ", event)

    # code to read uploaded image from bucket and create a ds object 

    # code below is never executed since the error appears when the cv2 library is loaded  

    img = cv2.resize(ds.pixel_array, (224, 224))

我的requirements.txt内容如下:

And my requirements.txt contents is as follows:

numpy==1.18.4
opencv-python==4.2.0.34
pandas==1.0.3
pydicom==1.4.2
python-dateutil==2.8.1
pytz==2018.7
six==1.14.0

我的问题是.如何使用插件将opencv依赖项正确上传到我的lambda函数?我应该以其他方式执行此操作吗?

My question is. How can I properly upload opencv dependencies to my lambda function using the plugin? Should I do this in another way?

谢谢！

推荐答案

基于

Based on the doc, you need to include those manually using dockerExtraFiles option.

=======================更新=======================

======================= UPDATE =======================

打包Python依赖关系并为AWS Lambda构建部署程序包可能非常棘手.深入研究后，需要进行以下操作才能使其正常运行.

Packaging Python dependencies and building deployment package for AWS Lambda can be quite tricky. After digging around, here are the things that you need to do to make it work.

FROM lambci/lambda:build-python3.8
RUN yum -y install libXext libSM libXrender

步骤2:修改 serverless.yml

service: sls-covid

provider:
  name: aws
  runtime: python3.8
  profile: testuser
  stage: ${opt:stage, 'staging'}
  environment: ${file(environment.yml):${self:provider.stage}}
  region: ${self:provider.environment.REGION}

  iamRoleStatements:
    # para poder leer y escribir en el bucket
    - Effect: "Allow"
      Action:
        - "s3:*"
      Resource: "*"

custom:
  pythonRequirements:
    dockerizePip: non-linux
    ########### BEGIN ###########
    dockerFile: Dockerfile
    dockerExtraFiles:
      - /lib64/libgthread-2.0.so.0
      - /lib64/libglib-2.0.so.0
      - /lib64/libSM.so.6
      - /lib64/libICE.so.6
      - /lib64/libXrender.so.1
      - /lib64/libXext.so.6
      - /lib64/libX11.so.6
      - /lib64/libuuid.so.1
      - /lib64/libxcb.so.1
      - /lib64/libXau.so.6
    ########### END ###########


package:
  individually: true
  exclude:
    - node_modules/**
    - env/**
    - package.json
    - package-lock.json

functions:

  analyzeImage:
    # para analizar la imagen cuando se almacena en S3
    handler: handlers.image.analyze
    events:
      - s3:
          bucket: ${self:service}-${self:provider.stage}-images
          event: s3:ObjectCreated:*
          rules:
            - prefix: uploads/

plugins:
  - serverless-python-requirements

第3步:Hello world测试

$ sls invoke -f analyzeImage --log
null
--------------------------------------------------------------------
START RequestId: 00ccd940-cf8a-46ed-8671-65e597f997a0 Version: $LATEST
=== event:  {}
END RequestId: 00ccd940-cf8a-46ed-8671-65e597f997a0
REPORT RequestId: 00ccd940-cf8a-46ed-8671-65e597f997a0  Duration: 1.41 ms   Billed Duration: 100 ms Memory Size: 1024 MB    Max Memory Used: 121 MB Init Duration: 916.32 ms

希望有帮助.

这篇关于Python中的AWS Lambda和Open CV的无服务器问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！