AWS Lambda中的Cassandra数据库会话重用(python) [英] Cassandra database session reuse in AWS Lambda (python)

问题描述

我正在尝试将Cassandra群集会话重用于后续的AWS Lambda函数调用.我已经在Java中成功实现了它，但是在python中重用会话会导致lambda调用超时(实际上执行初始化的第一个调用是可以的).

I am trying to reuse a Cassandra cluster session for subsequent AWS Lambda function calls. I've successfully implemented it in Java, but reusing the session in the python gets the lambda invocation timed out (the first call which actually performs the initialization is ok).

从CloudWatch日志中，我可以看到我得到了Heartbeat failed for connection.在我看来，会话在空闲时无法通信，并且处于不一致状态，无法恢复连接.实际上，尝试将idle_heartbeat_interval设置为比函数超时更长或更短的时间对结果没有任何影响.

From the CloudWatch logs I can see I get a Heartbeat failed for connection. It looks to me that the session is not able to communicate while idle, and that it gets in an inconsistent state in which is not able to resume the connection. Trying longer or shorter idle_heartbeat_interval than the function timeout in fact doesn't have any impact on the outcome.

这是我的lambda函数的结构(为简洁起见，省略了一些代码):

Here is the structure of my lambda function (omitted some code for brevity):

import logging
from cassandra_client import CassandraClient

logger = logging.getLogger()
logger.setLevel(logging.INFO)

#   State of the initialization phase
flag = False

#   Cassandra instance
cassandra = None

def handle_request(event, context):

    global flag, logger, cassandra

    logger.info('Function started. Flag: %s' % (str(flag), ))

    if not flag:
        logger.info('Initialization...')
        try:
            cassandra = CassandraClient()

            #   ...

            flag = True

        except Exception as e:
            logger.error('Cannot perform initialization: '+e.message)
            exit(-1)

    #   Process the request ...
    return 'OK'

为了完整起见，这就是我创建与群集的连接的方式:

Just for completeness, that's how I create the connection with the cluster:

def _connect(self, seed_nodes=default_seed_nodes, port=default_port):
    self.cluster = Cluster(seed_nodes, port=port)
    self.metadata = self.cluster.metadata
    self.session = self.cluster.connect()
    # ...

是否有一些驱动程序配置详细信息，我不知道的python lambda行为会阻止会话重用?

Is there some driver configuration detail, python lambda behavior I am not aware of that prevents the session to be reused?

我确实认为AWS Lambda是一个非常出色的工具，但是对执行没有太多控制可能会在某些方面造成混淆.任何建议都非常感谢，谢谢.

I do think AWS Lambda is a really great tool, but not having much control over the execution can be somehow confusing for certain aspects. Any suggestion is really appreciated, thanks.

推荐答案

我想我可以说这个问题是由使用Python执行环境w.r.t时lambda的不同行为引起的. Java.

I think I can state that this issue is cause by a different behavior of lambda when using the Python execution environment w.r.t. Java.

我有时间设置一个简单的lambda函数，在Java ad Python中都实现了.该函数只是产生一个线程，该线程在while循环中打印当前时间.问题是:即使lambda函数返回后，Java实现中的线程是否仍会继续打印，相反，Python线程会停止吗?在两种情况下答案都是肯定的:java线程将继续打印直到配置了超时，而python将在lambda函数返回时立即停止.

I had time to set up a simple lambda function, implemented both in Java ad Python. The function simply spawns a thread which prints the current time in a while loop. The question was: will the thread in the Java implementation continue printing even after the lambda function has returned, and conversely, will the Python thread stop instead? The answer is yes in both cases: the java thread continues printing till the timeout configured, while python will stop as soon as the lambda function returns.

Java版本的CloudWatch日志确认:

The CloudWatch log for the Java version confirms that:

09:55:21 START RequestId: b70e732b-e476-11e6-b2bb-e11a0dd9b311 Version: $LATEST
09:55:21 Function started: 1485510921351
09:55:21 Pre function call: 1485510921351
09:55:21 Background function: 1485510921352
09:55:21 Background function: 1485510921452
09:55:21 Background function: 1485510921552
09:55:21 Background function: 1485510921652
09:55:21 Background function: 1485510921752
09:55:21 Post function call: 1485510921852
09:55:21 Background function: 1485510921853
09:55:21 END RequestId: b70e732b-e476-11e6-b2bb-e11a0dd9b311
09:55:21 REPORT RequestId: b70e732b-e476-11e6-b2bb-e11a0dd9b311 Duration: 523.74 ms Billed Duration: 600 ms Memory Size: 256 MB Max Memory Used: 31 MB
09:55:21 Background function: 1485510921953
09:55:22 Background function: 1485510922053
...

在Python版本中:

While in the Python version:

09:01:04 START RequestId: 21ccc71e-e46f-11e6-926b-6b46f85c9c69 Version: $LATEST
09:01:04 Function started: 2017-01-27 09:01:04.189819
09:01:04 Pre function call: 2017-01-27 09:01:04.189849
09:01:04 background_call function: 2017-01-27 09:01:04.194368
09:01:04 background_call function: 2017-01-27 09:01:04.294617
09:01:04 background_call function: 2017-01-27 09:01:04.394843
09:01:04 background_call function: 2017-01-27 09:01:04.495100
09:01:04 background_call function: 2017-01-27 09:01:04.595349
09:01:04 Post function call: 2017-01-27 09:01:04.690483
09:01:04 END RequestId: 21ccc71e-e46f-11e6-926b-6b46f85c9c69
09:01:04 REPORT RequestId: 21ccc71e-e46f-11e6-926b-6b46f85c9c69 Duration: 500.99 ms Billed Duration: 600 ms Memory Size: 128 MB Max Memory Used: 8 MB

这是两个函数的代码:

Python

import thread
import datetime
import time


def background_call():
    while True:
        print 'background_call function: %s' % (datetime.datetime.now(), )
        time.sleep(0.1)

def lambda_handler(event, context):
    print 'Function started: %s' % (datetime.datetime.now(), )

    print 'Pre function call: %s' % (datetime.datetime.now(), )
    thread.start_new_thread(background_call, ())
    time.sleep(0.5)
    print 'Post function call: %s' % (datetime.datetime.now(), )

    return 'Needs more cowbell!'

Java

import com.amazonaws.services.lambda.runtime.*;


public class BackgroundTest implements RequestHandler<RequestClass, ResponseClass> {

    public static void main( String[] args )
    {
        System.out.println( "Hello World!" );
    }

    public ResponseClass handleRequest(RequestClass requestClass, Context context) {
        System.out.println("Function started: "+System.currentTimeMillis());
        System.out.println("Pre function call: "+System.currentTimeMillis());
        Runnable r = new Runnable() {
            public void run() {
                while(true){
                    try {
                        System.out.println("Background function: "+System.currentTimeMillis());
                        Thread.sleep(100);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                }
            }
        };
        Thread t = new Thread(r);
        t.start();
        try {
            Thread.sleep(500);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        System.out.println("Post function call: "+System.currentTimeMillis());
        return new ResponseClass("Needs more cowbell!");
    }
}

这篇关于AWS Lambda中的Cassandra数据库会话重用(python)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！