使用带有 Lambda 的 psycopg2 更新 Redshift (Python) [英] Using psycopg2 with Lambda to Update Redshift (Python)
问题描述
我正在尝试使用 python 从 Lambda 函数更新 Redshift.为此，我尝试组合 2 个代码片段.当我单独运行它们时，这两个片段都可以正常工作.
I am attempting to update Redshift from a Lambda function using python. To do this, I am attempting to combine 2 code fragments. Both fragments are functional when I run them separately.
从 PyDev 为 Eclipse 更新 Redshift
Updating Redshift from PyDev for Eclipse
import psycopg2
conn_string = "dbname='name' port='0000' user='name' password='pwd' host='url'"
conn = psycopg2.connect(conn_string)
cursor = conn.cursor()
cursor.execute("UPDATE table SET attribute='new'")
conn.commit()
cursor.close()
接收上传到 S3 存储桶的内容(Lambda 上可用的预构建模板)
Receiving Content Uploaded to S3 Bucket (Pre-Built Template Available on Lambda)
from __future__ import print_function
import json
import urllib
import boto3
print('Loading function')
s3 = boto3.client('s3')
def lambda_handler(event, context):
#print("Received event: " + json.dumps(event, indent=2))
# Get the object from the event and show its content type
bucket = event['Records'][0]['s3']['bucket']['name']
key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key']).decode('utf8')
try:
response = s3.get_object(Bucket=bucket, Key=key)
print("CONTENT TYPE: " + response['ContentType'])
return response['ContentType']
except Exception as e:
print(e)
print('Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.'.format(key, bucket))
raise e
由于这两个部分都有效，我尝试将它们组合起来，以便在将文件上传到 s3 时更新 Redshift:
Since both of these segments worked, I tried to combine them so that I could update Redshift upon the upload of a file to s3:
from __future__ import print_function
import json
import urllib
import boto3
import psycopg2
print('Loading function')
s3 = boto3.client('s3')
def lambda_handler(event, context):
#print("Received event: " + json.dumps(event, indent=2))
# Get the object from the event and show its content type
bucket = event['Records'][0]['s3']['bucket']['name']
key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key']).decode('utf8')
conn_string = "dbname='name' port='0000' user='name' password='pwd' host='url'"
conn = psycopg2.connect(conn_string)
cursor = conn.cursor()
cursor.execute("UPDATE table SET attribute='new'")
conn.commit()
cursor.close()
try:
response = s3.get_object(Bucket=bucket, Key=key)
print("CONTENT TYPE: " + response['Body'].read())
return response['Body'].read()
except Exception as e:
print(e)
print('Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.'.format(key, bucket))
raise e
由于我使用的是外部库，因此需要创建一个部署包.我创建了一个新文件夹 (lambda_function1) 并将我的 .py 文件 (lambda_function1.py) 移动到该文件夹.我运行以下命令在该文件夹中安装 psycopg2:
Since I am using an outside library, I need to create a deployment package. I created a new folder (lambda_function1) and moved my .py file (lambda_function1.py) to that folder. I ran the following command to install psycopg2 in that folder:
pip install psycopg2 -t lambda_function1
我收到以下反馈:
Collecting psycopg2
Using cached psycopg2-2.6.1-cp34-none-win_amd64.whl
Installing collected packages: psycopg2
Successfully installed psycopg2-2.6.1
然后我压缩了目录的内容.并将该 zip 上传到我的 lambda 函数.当我将文档上传到函数监控的存储桶时，我在 Cloudwatch 日志中收到以下错误:
I then zipped the contents of the directory. And uploaded that zip to my lambda function. When I upload a document to the bucket the function monitors, I receive the following error in my cloudwatch log:
Unable to import module 'lambda_function1': No module named _psycopg
当我查看图书馆时，唯一名为_psycopg"的东西是_psycopg.pyd".
When I look in the library, the only thing named "_psycopg" is "_psycopg.pyd".
是什么导致了这个问题?当我使用 3.4 时，Lambda 使用 Python 2.7 有关系吗?我在 Windows 机器上压缩文件的内容有关系吗?有没有人能够从 lambda 成功连接到 Redshift?
What is causing this problem? Does it matter that Lambda uses Python 2.7 when I use 3.4? Does it matter that I zipped the contents of my file on a Windows machine? Has anyone been able to successfully connect to Redshift from lambda?
推荐答案
为了让这个工作，你需要用静态链接的 libpq.so 库构建 psycopg2.查看此存储库 https://github.com/jkehler/awslambda-psycopg2.它已经构建了 psycopg2 包以及如何自己构建它的说明.
In order for this to work you need to build psycopg2 with statically linked libpq.so library. Check out this repo https://github.com/jkehler/awslambda-psycopg2. It has already build psycopg2 package and instructions how to build it yourself.
回到你的问题:
是什么导致了这个问题?
psycopg2 需要为 Linux 构建一个使用静态链接库编译的.
psycopg2 needs to be build an compiled with statically linked libraries for Linux.
当我使用 3.4 时，Lambda 使用 Python 2.7 有关系吗?
是的，lambda 只支持 2.7 版本.只需创建虚拟环境并在其中安装所有必要的包.
Yes it does, lambda only supports 2.7 version. Just create virtual environment and install all necessary packages in there.
我在 Windows 机器上压缩文件内容有关系吗?
只要你压缩的所有库都可以在 Linux 上运行，它就不会
As long as all the libraries you zipped could ran on Linux it doesn't
有没有人能够成功地从 lambda 连接到 Redshift?
是的.
这篇关于使用带有 Lambda 的 psycopg2 更新 Redshift (Python)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
