如何在cloudfoundry上使用kafka和storm? [英] How to use kafka and storm on cloudfoundry?

问题描述

我想知道是否可以将 kafka 作为云原生应用程序运行，我是否可以在 Pivotal Web Services 上创建 kafka 集群作为服务.我不想只做客户端集成，我想自己运行 kafka 集群/服务?

谢谢，阿尼尔

解决方案

我可以指出几个起点，从这些起点到功能齐全的东西需要一些工作.

一种选择是使用 docker 镜像在 Cloud Foundry(例如 Pivotal Web 服务)上部署 kafka 集群.Spotify 有 Dockerized kafka 和 kafka-proxy(包括 Zookeeper).要记住的一件事是 PWS 目前不支持具有持久性的应用程序(尽管 这项工作正在开始) 所以如果你现在走这条路，你会在应用程序滚动时丢失 kafka 中的数据.看看那个 Spotify 存储库，看起来 docker 镜像通常在没有任何挂载卷的情况下运行，所以这个无持久性的 kafka 似乎是一个有效的用例(我对 kafka 不太了解).

另一种选择是使用 BOSH 将 kafka 直接部署在某些 IaaS(例如 AWS)上.如果您是第一次看到 BOSH，它可能会很难，但它是部署您想要在 VM 上运行的任何分布式软件的理想方式.如有必要，您还可以将持久卷附加到您的 kafka 虚拟机.这是一个 kafka BOSH 版本，可能会起作用.

集群运行后，您可以通过两种方式将 Cloud Foundry 应用程序与其集成.最简单的方法就是将其作为用户提供的服务"，它允许您将 kafka 集群访问信息传输到您的应用程序.另一种方法是在您的集群前面放置一个服务代理，如果您有许多不同的人将推送需要与 kafka 集群通信的应用程序，这将特别有用.您不必每次都手动告诉人们访问信息，他们可以做一些简单的事情，例如 cf bind-service SOME_APP YOUR_KAFKA_SERVICE.这是一个 kafka 服务代理以及 有关服务代理的一般信息.

I want to know if it is possible to run kafka as a cloud-native application, and can I create a kafka cluster as a service on Pivotal Web Services. I don't want only client integration, I want to run the kafka cluster/service itself?

Thanks, Anil

解决方案

I can point you at a few starting points, there would be some work involved to go from those starting points to something fully functional.

One option is to deploy the kafka cluster on Cloud Foundry (e.g. Pivotal Web Services) using docker images. Spotify has Dockerized kafka and kafka-proxy (including Zookeeper). One thing to keep in mind is that PWS currently doesn't support apps with persistence (although this work is starting) so if you were to go this route right now, you would lose the data in kafka when the application is rolled. Looking at that Spotify repo, it looks like the docker images are generally run without any mounted volumes, so this persistence-less kafka seems like it may be a valid use case (I don't know enough about kafka to say).

The other option is to deploy kafka directly on some IaaS (e.g. AWS) using BOSH. BOSH can be hard if you're seeing it for the first time, but it is the ideal way to deploy any distributed software that you want running on VMs. You will also be able to have persistent volumes attached to your kafka VMs if necessary. Here is a kafka BOSH release which may work.

Once you have your cluster running, you have two ways to integrate your Cloud Foundry applications with it. The simplest is just to provide it to your applications as a "user-provided service", which lets you flow kafka cluster access info to your apps. The alternative would to put a service broker in front of your cluster, which would be especially useful if you have many different people who will be pushing apps that need to talk to the kafka cluster. Rather than you having to manually tell people the access info each time, they can do something simple like cf bind-service SOME_APP YOUR_KAFKA_SERVICE. Here is a kafka service broker along with more info about service brokers in general.

这篇关于如何在cloudfoundry上使用kafka和storm?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！