将psycopg2与Lambda一起使用以更新Redshift(Python) [英] Using psycopg2 with Lambda to Update Redshift (Python)
问题描述
我正在尝试使用python从Lambda函数更新Redshift.为此，我尝试合并2个代码片段.当我分别运行它们时，这两个片段都可以正常工作.
I am attempting to update Redshift from a Lambda function using python. To do this, I am attempting to combine 2 code fragments. Both fragments are functional when I run them separately.
-
从PyDev for Eclipse更新Redshift
Updating Redshift from PyDev for Eclipse
import psycopg2
conn_string = "dbname='name' port='0000' user='name' password='pwd' host='url'"
conn = psycopg2.connect(conn_string)
cursor = conn.cursor()
cursor.execute("UPDATE table SET attribute='new'")
conn.commit()
cursor.close()
接收上载到S3存储桶的内容(Lambda上提供预构建的模板)
Receiving Content Uploaded to S3 Bucket (Pre-Built Template Available on Lambda)
from __future__ import print_function
import json
import urllib
import boto3
print('Loading function')
s3 = boto3.client('s3')
def lambda_handler(event, context):
#print("Received event: " + json.dumps(event, indent=2))
# Get the object from the event and show its content type
bucket = event['Records'][0]['s3']['bucket']['name']
key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key']).decode('utf8')
try:
response = s3.get_object(Bucket=bucket, Key=key)
print("CONTENT TYPE: " + response['ContentType'])
return response['ContentType']
except Exception as e:
print(e)
print('Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.'.format(key, bucket))
raise e
由于这两个段都起作用，所以我尝试将它们组合在一起，以便在将文件上传到s3时可以更新Redshift:
Since both of these segments worked, I tried to combine them so that I could update Redshift upon the upload of a file to s3:
from __future__ import print_function
import json
import urllib
import boto3
import psycopg2
print('Loading function')
s3 = boto3.client('s3')
def lambda_handler(event, context):
#print("Received event: " + json.dumps(event, indent=2))
# Get the object from the event and show its content type
bucket = event['Records'][0]['s3']['bucket']['name']
key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key']).decode('utf8')
conn_string = "dbname='name' port='0000' user='name' password='pwd' host='url'"
conn = psycopg2.connect(conn_string)
cursor = conn.cursor()
cursor.execute("UPDATE table SET attribute='new'")
conn.commit()
cursor.close()
try:
response = s3.get_object(Bucket=bucket, Key=key)
print("CONTENT TYPE: " + response['Body'].read())
return response['Body'].read()
except Exception as e:
print(e)
print('Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.'.format(key, bucket))
raise e
由于我使用的是外部库，因此需要创建一个部署程序包.我创建了一个新文件夹(lambda_function1)并将我的.py文件(lambda_function1.py)移至该文件夹.我运行以下命令在该文件夹中安装psycopg2:
Since I am using an outside library, I need to create a deployment package. I created a new folder (lambda_function1) and moved my .py file (lambda_function1.py) to that folder. I ran the following command to install psycopg2 in that folder:
pip install psycopg2 -t \lambda_function1
我收到以下反馈:
Collecting psycopg2
Using cached psycopg2-2.6.1-cp34-none-win_amd64.whl
Installing collected packages: psycopg2
Successfully installed psycopg2-2.6.1
然后我压缩目录的内容.并将该zip文件上传到我的lambda函数中.当我将文档上传到功能监控的存储桶时，我的cloudwatch日志中收到以下错误:
I then zipped the contents of the directory. And uploaded that zip to my lambda function. When I upload a document to the bucket the function monitors, I receive the following error in my cloudwatch log:
Unable to import module 'lambda_function1': No module named _psycopg
当我在库中查看时，唯一名为"_psycopg"的是"_psycopg.pyd".
When I look in the library, the only thing named "_psycopg" is "_psycopg.pyd".
是什么导致此问题?当我使用3.4时，Lambda使用Python 2.7是否重要?在Windows计算机上压缩文件内容是否重要?有没有人能够从lambda成功连接到Redshift?
What is causing this problem? Does it matter that Lambda uses Python 2.7 when I use 3.4? Does it matter that I zipped the contents of my file on a Windows machine? Has anyone been able to successfully connect to Redshift from lambda?
推荐答案
为此，您需要使用静态链接的libpq.so库构建psycopg2.查看此回购 https://github.com/jkehler/awslambda-psycopg2 .它已经构建了psycopg2软件包并说明了如何自行构建.
In order for this to work you need to build psycopg2 with statically linked libpq.so library. Check out this repo https://github.com/jkehler/awslambda-psycopg2. It has already build psycopg2 package and instructions how to build it yourself.
回到您的问题:
是什么原因导致此问题?
psycopg2需要使用用于Linux的静态链接库来构建.
psycopg2 needs to be build an compiled with statically linked libraries for Linux.
当我使用3.4时Lambda使用Python 2.7是否重要?
是的，lambda仅支持2.7版本.只需创建虚拟环境并在其中安装所有必需的软件包即可.
Yes it does, lambda only supports 2.7 version. Just create virtual environment and install all necessary packages in there.
在Windows计算机上压缩文件内容是否重要?
只要您压缩的所有库都可以在Linux上运行
As long as all the libraries you zipped could ran on Linux it doesn't
有人能从lambda成功连接到Redshift吗?
是.
这篇关于将psycopg2与Lambda一起使用以更新Redshift(Python)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
