当在本地运行时，AWS Lambda Function可以导入模块，但在部署时不能导入 [英] AWS Lambda Function can import a module when run locally, but not when deployed

问题描述

我正在尝试在本指南，方法是构建一个CodePipeline以在GitHub中拾取更改，然后进行构建并将更改部署到我的Lambda中. sam build --use-container; sam local start-api允许我在本地成功调用该函数-但是，当我将该函数部署到AWS时，代码无法导入依赖项.

我的代码取决于requests.我已将其适当地包含在我的requirements.txt文件中:

requests==2.20.0

我的buildspec.yml包括安装依赖项的说明

version: 0.1
phases:
  install:
    commands:
      - pip install -r hello_world/requirements.txt -t .
      - pip install -U pytest
  pre_build:
    commands:
      - python -m pytest tests/
  build:
    commands:
      - aws cloudformation package --template-file template.yaml --s3-bucket <my_bucket>
                                   --output-template-file outputTemplate.yml
artifacts:
  type: zip
  files:
    - '**/*'

当我的程序包在CodeBuild中构建时，将被确认:

[Container] 2018/12/27 23:16:44 Waiting for agent ping 
[Container] 2018/12/27 23:16:46 Waiting for DOWNLOAD_SOURCE 
[Container] 2018/12/27 23:16:46 Phase is DOWNLOAD_SOURCE 
[Container] 2018/12/27 23:16:46 CODEBUILD_SRC_DIR=/codebuild/output/src775882062/src 
[Container] 2018/12/27 23:16:46 YAML location is /codebuild/output/src775882062/src/buildspec.yml 
[Container] 2018/12/27 23:16:46 Processing environment variables 
[Container] 2018/12/27 23:16:46 Moving to directory /codebuild/output/src775882062/src 
[Container] 2018/12/27 23:16:46 Registering with agent 
[Container] 2018/12/27 23:16:46 Phases found in YAML: 3 
[Container] 2018/12/27 23:16:46  PRE_BUILD: 1 commands 
[Container] 2018/12/27 23:16:46  BUILD: 1 commands 
[Container] 2018/12/27 23:16:46  INSTALL: 2 commands 
[Container] 2018/12/27 23:16:46 Phase complete: DOWNLOAD_SOURCE Success: true 
[Container] 2018/12/27 23:16:46 Phase context status code:  Message:  
[Container] 2018/12/27 23:16:46 Entering phase INSTALL 
[Container] 2018/12/27 23:16:46 Running command pip install -r hello_world/requirements.txt -t . 
Collecting requests==2.20.0 (from -r hello_world/requirements.txt (line 1)) 
  Downloading https://files.pythonhosted.org/packages/f1/ca/10332a30cb25b627192b4ea272c351bce3ca1091e541245cccbace6051d8/requests-2.20.0-py2.py3-none-any.whl (60kB)
...

但是当我调用已部署的函数时，出现错误:

Unable to import module 'app': No module named 'requests'

这似乎与此问题非常相似，但是我不在Lambda大楼中使用PYTHONPATH.

---

我向此软件包中的文件添加了一些调试代码，以试图了解它们的运行时环境.我还为通过CodePipeline部署到Lambda的另一种软件包添加了类似的调试功能(尽管该工具没有(不使用SAM).调试代码如下:

import os, sys
print('Inside ' + __file__)
for path in sys.path:
    print(path)
    if (os.path.exists(path)):
        print(os.listdir(path))
        for f in os.listdir(path):
          if f.startswith('requests'):
            print('Found requests!')
    print()

此代码尝试确定Lambda的运行时环境的sys.path中是否存在requests模块-如果存在，则在何处.

对于此(启用了SAM的)程序包，在任何地方都找不到requests.在未启用SAM的程序包中，在/var/task中找到了requests(以及该程序包的所有其他requirements.txt声明的依赖项).

似乎CodeBuild没有将函数的依赖项与源代码捆绑在一起，或者CloudFormation没有部署这些依赖项.我怀疑这与以下事实有关:这是SAM定义的函数，而不是原始的" Cloudformation函数.

此页面说您还可以使用与AWS SAM集成的其他AWS服务来自动化您的部署"，但我看不到如何让CodePipeline运行sam deploy而不是aws cloudformation deploy(尽管或...BucketName"]}获取)

以下是第一个包的CodeBuild输出的示例:

[Container] 2018/12/30 19:19:48 Running command aws cloudformation package --template-file template.yaml --s3-bucket pop-culture-serverless-bucket --output-template-file outputTemplate.yml 

Uploading to 669099ba3d2258eeb7391ad772bf870d  4222 / 4222.0  (100.00%) 
Successfully packaged artifacts and wrote output template to file outputTemplate.yml. 
Execute the following command to deploy the packaged template 
aws cloudformation deploy --template-file /codebuild/output/src110881899/src/outputTemplate.yml --stack-name <YOUR STACK NAME> 

与第二个程序包的CodeBuild输出中的相同输出进行比较:

....
[Container] 2018/12/30 16:42:27 Running command aws cloudformation package --template-file template.json --s3-bucket {BUCKET_NAME} --output-template-file outputTemplate.yml 

Successfully packaged artifacts and wrote output template to file outputTemplate.yml. 
Execute the following command to deploy the packaged template 
aws cloudformation deploy --template-file /codebuild/output/src282566886/src/outputTemplate.yml --stack-name <YOUR STACK NAME>

这表明第一个程序包的aws cloudformation package调用导致将文件(669099ba3d2258eeb7391ad772bf870d)上载到S3，该文件基于的内容仅 ，而第二个程序包的CodePipeline的Build阶段的输出"是CodeBuild已在 in 中运行的目录的zip-包括依赖项(由于对pip install的调用).

我可以通过简单地更改我的SAM模板功能的template.yaml以引用S3位置来解决此问题-但这意味着我将无法在本地测试对该功能的更新(例如sam local start-api)，而无需编辑模板，因为它会引用S3位置，因此不会受到本地更改的影响.

理想情况下，我想找到一种在打包并上传的S3文件中包含代码依赖项的方法.从本地测试看来，运行sam package/aws cloudformation package而不先运行sam build会导致仅包含源代码(不存在依赖项).但是，由于未在其中安装SAM，因此无法在CodeBuild中运行sam build.

(这也表明我无意中部署了第二个程序包的测试相关性-因为需要将它们安装在CodeBuild中(以便运行测试))

解决方案

通过在main目录而不是根目录中安装我的代码的依存关系，我找到了一个解决方案".但是，我认为一种更好的选择是使用 layers 保留依赖项.

I am attempting to expand on this guide, by building a CodePipeline to pick up changes in GitHub, build them, and deploy the changes to my Lambda. sam build --use-container; sam local start-api allows me to successfully call the function locally - but when I deploy the function to AWS, the code fails to import a dependency.

My code depends on requests. I have duly included that in my requirements.txt file:

requests==2.20.0

My buildspec.yml includes directions to install the dependencies

version: 0.1
phases:
  install:
    commands:
      - pip install -r hello_world/requirements.txt -t .
      - pip install -U pytest
  pre_build:
    commands:
      - python -m pytest tests/
  build:
    commands:
      - aws cloudformation package --template-file template.yaml --s3-bucket <my_bucket>
                                   --output-template-file outputTemplate.yml
artifacts:
  type: zip
  files:
    - '**/*'

When my package builds in CodeBuild, that is acknowledged:

[Container] 2018/12/27 23:16:44 Waiting for agent ping 
[Container] 2018/12/27 23:16:46 Waiting for DOWNLOAD_SOURCE 
[Container] 2018/12/27 23:16:46 Phase is DOWNLOAD_SOURCE 
[Container] 2018/12/27 23:16:46 CODEBUILD_SRC_DIR=/codebuild/output/src775882062/src 
[Container] 2018/12/27 23:16:46 YAML location is /codebuild/output/src775882062/src/buildspec.yml 
[Container] 2018/12/27 23:16:46 Processing environment variables 
[Container] 2018/12/27 23:16:46 Moving to directory /codebuild/output/src775882062/src 
[Container] 2018/12/27 23:16:46 Registering with agent 
[Container] 2018/12/27 23:16:46 Phases found in YAML: 3 
[Container] 2018/12/27 23:16:46  PRE_BUILD: 1 commands 
[Container] 2018/12/27 23:16:46  BUILD: 1 commands 
[Container] 2018/12/27 23:16:46  INSTALL: 2 commands 
[Container] 2018/12/27 23:16:46 Phase complete: DOWNLOAD_SOURCE Success: true 
[Container] 2018/12/27 23:16:46 Phase context status code:  Message:  
[Container] 2018/12/27 23:16:46 Entering phase INSTALL 
[Container] 2018/12/27 23:16:46 Running command pip install -r hello_world/requirements.txt -t . 
Collecting requests==2.20.0 (from -r hello_world/requirements.txt (line 1)) 
  Downloading https://files.pythonhosted.org/packages/f1/ca/10332a30cb25b627192b4ea272c351bce3ca1091e541245cccbace6051d8/requests-2.20.0-py2.py3-none-any.whl (60kB)
...

But when I call the deployed function, I get an error:

Unable to import module 'app': No module named 'requests'

This seems very similar to this question, but I'm not using PYTHONPATH in my Lambda building.

---

EDIT: I added some debugging code to files in this package, to try to get a sense of their runtime environment. I also added similar debugging to another package that I deploy to Lambda via CodePipeline (though this one doesn't use SAM). Debugging code is below:

import os, sys
print('Inside ' + __file__)
for path in sys.path:
    print(path)
    if (os.path.exists(path)):
        print(os.listdir(path))
        for f in os.listdir(path):
          if f.startswith('requests'):
            print('Found requests!')
    print()

This code attempts to determine if the requests module is present in the sys.path of the Lambda's runtime environment - and, if so, where.

For this (SAM-enabled) package, requests was not found anywhere. In the non-SAM-enabled package, requests (as well as all the other requirements.txt-declared dependencies of the package) was found in /var/task.

It looks like either CodeBuild isn't bundling the dependencies of the function alongside the source, or CloudFormation isn't deploying those dependencies. I suspect that this is something to do with the fact that this is a SAM-defined function, and not a "vanilla" Cloudformation one.

This page says that "You can also use other AWS services that integrate with AWS SAM to automate your deployments", but I can't see how to get CodePipeline to run sam deploy instead of aws cloudformation deploy (although this page claims that they are synonyms).

---

EDIT2 - I believe I've found the problem. For context, recall that I have two packages that are deploying Lambdas via CodePipeline (or attempting to) - the one referred to in this question, which refers to the Lambda as AWS::Serverless::Function, and a second, which uses AWS::Lambda::Function. The first Function's code was defined as a relative location (i.e., a reference to a directory in my package: CodeUri: main/), whereas the second Function's Code was a reference to an S3 location (fetched, in CodePipeline, with Fn::GetArtifactAtt": ["built", "ObjectKey"]} or ...BucketName"]})

The following is a sample of the first package's CodeBuild output:

[Container] 2018/12/30 19:19:48 Running command aws cloudformation package --template-file template.yaml --s3-bucket pop-culture-serverless-bucket --output-template-file outputTemplate.yml 

Uploading to 669099ba3d2258eeb7391ad772bf870d  4222 / 4222.0  (100.00%) 
Successfully packaged artifacts and wrote output template to file outputTemplate.yml. 
Execute the following command to deploy the packaged template 
aws cloudformation deploy --template-file /codebuild/output/src110881899/src/outputTemplate.yml --stack-name <YOUR STACK NAME> 

Compare with the same output from the second package's CodeBuild output:

....
[Container] 2018/12/30 16:42:27 Running command aws cloudformation package --template-file template.json --s3-bucket {BUCKET_NAME} --output-template-file outputTemplate.yml 

Successfully packaged artifacts and wrote output template to file outputTemplate.yml. 
Execute the following command to deploy the packaged template 
aws cloudformation deploy --template-file /codebuild/output/src282566886/src/outputTemplate.yml --stack-name <YOUR STACK NAME>

This suggests that the first package's aws cloudformation package call results in an upload of a file (669099ba3d2258eeb7391ad772bf870d) to S3 which is based only on the content of the template.yaml, whereas the "output" of the Build stage of the second package's CodePipeline is a zip of the directory that CodeBuild has been running in - which includes the dependencies (because of the calls to pip install).

I could get around this by simply changing the template.yaml of my SAM-template Function to reference an S3 location - but this would mean that I would be unable to test updates to the function locally (with, e.g., sam local start-api) without editing the template, since it would reference the S3 location and so wouldn't be affected by local changes.

Ideally, I want to find a way to include the dependencies of the code in the packaged-and-uploaded S3 file. From local testing, it appears that running sam package/aws cloudformation package without having first run sam build results in just the source code (no dependencies) being included. However, I can't run sam build in CodeBuild, since SAM isn't installed there.

(This also suggests that I have been unintentionally deploying the test-dependencies of my second package - since it was required to install them in CodeBuild (in order to run tests))

解决方案

I found a "solution" to this, by installing the dependencies of my code in the main directory rather than the root directory. However, I believe that a superior option would be to use layers to hold dependencies.

这篇关于当在本地运行时，AWS Lambda Function可以导入模块，但在部署时不能导入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！