AWS lambda 中的 Pandas 给出了 numpy 错误 [英] Pandas in AWS lambda gives numpy error

问题描述

我一直在尝试在导入熊猫的 AWS Lambda 中运行我的代码.所以这就是我所做的.我有一个 python 文件，它包含一个简单的代码如下(这个文件有 lambda 处理程序)

导入jsonprint('加载函数')将熊猫导入为 pddef lambda_handler(事件，上下文):返回欢迎在 AWS Lambda 中使用 Pandas"

1. 我已将此 python 文件与 numpy、pandas 和 pytz 库一起压缩为部署包(所有这些都在 Amazon EC2 linux 机器中完成)
2. 然后将包上传到 S3
3. 创建了一个 lambda 函数(runtime=python3.6)并从 S3 上传了部署包

但是当我在 AWS Lambda 中测试 lambda 函数时，出现以下错误:

无法导入模块lambda_function":缺少必需的依赖项 ['numpy']

我的压缩包中已经有 numpy，但仍然出现此错误.我试图按照 Pandas & 中给出的提示进行操作.AWS Lambda 但没有运气.

有没有人遇到过同样的问题.感谢您提供解决此问题的任何提示或建议.

谢谢

解决方案

经过大量研究后，我使其能够与 Lambda 层一起使用.

创建或打开一个干净的目录并按照以下步骤操作:

先决条件:确保您已启动并运行 Docker

1. 使用以下内容创建 requirements.txt 文件:

<块引用>

pandas==0.23.4pytz==2018.7

2. 使用以下内容创建 get_layer_packages.sh 文件:

<块引用>

#!/bin/bashexport PKG_DIR="python";rm -rf ${PKG_DIR} &&mkdir -p ${PKG_DIR}docker run --rm -v $(pwd):/foo -w/foo Lambci/lambda:build-python3.6 pip install -r requirements.txt --no-deps -t ${PKG_DIR}

3. 在同一目录中运行以下命令:

<块引用>

chmod +x get_layer_packages.sh./get_layer_packages.shzip -r pandas.zip .

4. 将层上传到 S3 存储桶.

5. 通过运行以下命令将层上传到 AWS:

<块引用>

aws lambda publish-layer-version --layer-name pandas-layer --description 你的层的描述"--content S3Bucket=,S3Key=.zip--compatible-runtimes python3.6 python3.7

6. 转到 Lambda 控制台并将您的代码作为 zip 文件上传或使用内联编辑器.

7. 点击图层>添加图层>在 Compatible 层中搜索层(pandas-layer)并选择版本.

8. 还要添加 AWSLambda-Python36-SciPy1x 层，默认情况下该层可用于导入 numpy.

从控制台选择图层

9. 测试代码.它现在应该可以工作了！！！！

感谢这篇中等文章https://medium.com/@qtangs/creating-new-aws-lambda-layer-for-python-pandas-library-348b126e9f3e

I've been trying to run my code in AWS Lambda which imports pandas. So here is what I've done. I have a python file which contains a simple code as follows(This file has the lambda handler)

import json
print('Loading function')
import pandas as pd
def lambda_handler(event, context):
    return "Welcome to Pandas usage in AWS Lambda"

1. I have zipped this python file along with numpy, pandas and pytz libraries as a deployment package (Did all these in Amazon EC2 linux machine)
2. Then uploaded the package into S3
3. Created a lambda function(runtime=python3.6) and uploaded the deployment package from S3

But when I test the lambda function in AWS Lambda, I get the below error:

Unable to import module 'lambda_function': Missing required dependencies ['numpy']

I already have numpy in the zipped package but still I get this error. I tried to follow the hints given at Pandas & AWS Lambda but no luck.

Did anyone ran into the same issue. Would appreciate any hint or suggestions to solve this problem.

Thanks

解决方案

After doing a lot of research I was able to make it work with Lambda layers.

Create or open a clean directory and follow the steps below:

Prerequisites: Make sure you have Docker up and running

1. Create a requirements.txt file with the following:

pandas==0.23.4
pytz==2018.7

2. Create a get_layer_packages.sh file with the following:

#!/bin/bash

export PKG_DIR="python"

rm -rf ${PKG_DIR} && mkdir -p ${PKG_DIR}

docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.6 
    pip install -r requirements.txt --no-deps -t ${PKG_DIR}

3. Run the following commands in the same directory:

chmod +x get_layer_packages.sh

./get_layer_packages.sh

zip -r pandas.zip .

4. Upload the layer to a S3 bucket.

5. Upload the layer to AWS by running the command below:

aws lambda publish-layer-version --layer-name pandas-layer --description "Description of your layer"
--content S3Bucket=<bucket name>,S3Key=<layer-name>.zip
--compatible-runtimes python3.6 python3.7

6. Go to Lambda console and upload your code as a zip file or use the inline editor.

7. Click on Layers > Add a layer> Search for the layer (pandas-layer) from the Compatible layers and select the version.

8. Also add the AWSLambda-Python36-SciPy1x layer which is available by default for importing numpy.

Selecting the layer from the console

9. Test the code. It should work now!!!!

Thanks to this medium article https://medium.com/@qtangs/creating-new-aws-lambda-layer-for-python-pandas-library-348b126e9f3e

这篇关于AWS lambda 中的 Pandas 给出了 numpy 错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！