卡夫卡 + AWS 拉姆达 [英] Kafka + AWS lambda

问题描述

是否可以将 AWS Lambda 与 Apache Kafka 集成?我想将消费者放入 lambda 函数中.当消费者收到消息时，lambda 函数会执行.

Is it possible to integrate AWS Lambda with Apache Kafka ? I want to put a consumer in a lambda function. When a consumer receive a message the lambda function execute.

推荐答案

继续阿拉法特的观点.我们已经成功构建了一个基础设施，可以使用 AWS Lambdas 从 Kafka 消费.这里有一些问题:

Continuing the point by Arafat. We have successfully built an infrastructure to consume from Kafka using AWS Lambdas. Here are some gotcha's:

• 确保在阅读时始终如一地进行批处理和提交.
• 如果您要将批处理存储到 s3，请确保清理您的文件描述符.
• 如果您要将批次转发到另一个服务，请确保清除变量.AWS Lambda 中的变量缓存可能会导致内存溢出.
• 一个好主意是检查您从 Lambda 中的 context 对象还剩下多少时间，并给自己一些回旋余地来处理您在消费者中填充的缓冲区，这可能不会除非您调用 close()，否则将被读取到文件中.

• Make sure to consistently batch and commit while reading when consuming.
• If you are storing the batches to s3, make sure to clean your file descriptors.
• If you are forwarding the batches to another service make sure to clean the variables. Variable caching in AWS Lambda might result in memory overflows.
• A good idea is to check how much time you have left while from the context object in the Lambda and give yourself some wiggle room to do something with the buffer you populated in your consumer which might not be read to a file unless you call close().

我们使用 Apache Airflow 进行调度.我听说 cloudwatch 也可以.

We are using Apache Airflow for scheduling. I hear cloudwatch can do that too.

这篇关于卡夫卡 + AWS 拉姆达的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！