kubernetes中的两个单独的hazelcast集群 [英] Two separate hazelcast clusters in kubernetes

问题描述

为了在Kubernetes命名空间中的两个不同服务的pod之间共享事件，我打算使用Hazelcast.这不是问题，但是，每个服务还具有一个包含其所有Pod的集群.

For sharing events between the pods of two different services in a Kubernetes namespace, I intend to use Hazelcast. This is not a problem, however, each service also has a cluster that contains all its pods.

因此，我有两个使用相同吊舱的集群.我通过为其中一个集群设置组名来实现集群的分离，而另一个具有默认的组配置.这可以在本地正常运行，并具有测试应用程序的多个实例.但是，这是启用多播的.

So, I have two clusters using the same pods. I achieved separation of the clusters by setting a group name for one of the clusters, while the other has the default group configuration. This works fine locally, with multiple instances of a test application. However, this is with multicast enabled.

但是，在Kubernetes中，Hazelcast使用 HazelcastKubernetesDiscoveryStrategy ，并且已禁用多播.

In Kubernetes however, Hazelcast uses the HazelcastKubernetesDiscoveryStrategy and has Multicast disabled.

这两个服务都有一个标签:

Both services have a label:

metadata:
  name: service-1
  labels:
    hazelcast-group: bc-events

metadata:
  name: service-2
  labels:
    hazelcast-group: bc-events

事件集群的hazelcast配置如下:

and the hazelcast configuration for the events cluster is like this:

Config hzConfig = new Config("events-instance");

NetworkConfig nwConfig = new NetworkConfig();
JoinConfig joinConfig = new JoinConfig();
joinConfig.setMulticastConfig(new MulticastConfig().setEnabled(false));
joinConfig.setTcpIpConfig(new TcpIpConfig().setEnabled(false));

DiscoveryStrategyConfig k8sDiscoveryStrategy = new DiscoveryStrategyConfig("com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy");
k8sDiscoveryStrategy.addProperty("namespace", "dev");
k8sDiscoveryStrategy.addProperty("resolve-not-ready-addresses", true);
k8sDiscoveryStrategy.addProperty("service-label-name", "hazelcast-group");
k8sDiscoveryStrategy.addProperty("service-label-value", "bc-events");

DiscoveryConfig discoveryConfig = new DiscoveryConfig();
discoveryConfig.addDiscoveryStrategyConfig(k8sDiscoveryStrategy);
joinConfig.setDiscoveryConfig(discoveryConfig);
nwConfig.setJoin(joinConfig);

hzConfig.setNetworkConfig(nwConfig);
hzConfig.setProperty("hazelcast.discovery.enabled", "true");

GroupConfig groupConfig = new GroupConfig("bc-events");
hzConfig.setGroupConfig(groupConfig);

共享缓存群集(没有一组的缓存)的配置是这样的(对于服务1，服务2是相同的):

while the configuration for the shared cache cluster (the one without a group) is like this (for service 1, service 2 is the same):

Config hzConfig = new Config("service-1-app-hc");

NetworkConfig nwConfig = new NetworkConfig();
JoinConfig joinConfig = new JoinConfig();
joinConfig.setMulticastConfig(new MulticastConfig().setEnabled(false));
joinConfig.setTcpIpConfig(new TcpIpConfig().setEnabled(false));

DiscoveryStrategyConfig k8sDiscoveryStrategy = new DiscoveryStrategyConfig("com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy");

k8sDiscoveryStrategy.addProperty("namespace", "dev");
k8sDiscoveryStrategy.addProperty("service-name", "service-1");
k8sDiscoveryStrategy.addProperty("resolve-not-ready-addresses", true);

DiscoveryConfig discoveryConfig = new DiscoveryConfig();
discoveryConfig.addDiscoveryStrategyConfig(k8sDiscoveryStrategy);
joinConfig.setDiscoveryConfig(discoveryConfig);
nwConfig.setJoin(joinConfig);

hzConfig.setNetworkConfig(nwConfig);
hzConfig.setProperty("hazelcast.discovery.enabled", "true");

hazelcast实例会找到对方，但随后又抱怨对方具有不同的组名并将IP列入黑名单.

The hazelcast instances find eachother, but then complain that the other has a different group name and blacklists the IP.

在调试代码的同时，在处理连接请求的配置验证期间，它会尝试比较组名(bc-events与dev)，这显然是不同的.但这会被列入黑名单(我相信)，从而阻止对具有相同组名的另一个实例进行验证检查.

While debugging the code, during config validation while processing a join request, it tries to compare the group names (bc-events against dev) and obviously that's different. But then this gets blacklisted (I believe), preventing a validation check for the other instance that has the same group name.

我不确定下一步要去哪里.我无法在本地测试此配置，因为如果没有多播，它将找不到其他要加入集群的节点.我也认为配置没有任何问题.

I'm not sure where to go next. I cannot test this configuration locally because without multicast, it doesn't find the other nodes for joining the cluster. I also don't think there is anything wrong with the configuration.

使用的库为:

<dependency>
    <groupId>com.hazelcast</groupId>
    <artifactId>hazelcast</artifactId>
    <version>3.7.8</version>
</dependency>
<dependency>
    <groupId>com.hazelcast</groupId>
    <artifactId>hazelcast-kubernetes</artifactId>
    <version>1.1.0</version>
</dependency>

更新:

我应该注意，在当前设置下，具有组名的集群可以通过服务名称发现(因此，它仅包含一项服务的容器).没有组的群集和具有组的群集彼此并排运行.只有当我切换到基于标签的发现(并且涉及其他服务)时，它才会中断.

I should note that, the current setup, where this cluster that has the group name discovers by service name (as such, it only contains the pods of one service) actually works. The cluster without a group and the cluster with a group are running alongside eachother. It is only when I switch to label-based discovery (and that the other service gets involved) that it breaks.

更新:

当更改事件群集的端口时，我注意到它虽然放在5801上，但仍然尝试连接到5701(默认值).当然，这是有效的，因为第一个群集在5701上运行.在HazelcastKubernetesDiscoveryStrategy内部有以下方法:

When changing the port of the events cluster, I noticed it still tries to connect to 5701 (the default) despite being put on 5801. Of course this works because the first cluster is running on 5701. Inside HazelcastKubernetesDiscoveryStrategy there is the following method:

protected int getServicePort(Map<String, Object> properties) {
    int port = NetworkConfig.DEFAULT_PORT;
    if (properties != null) {
        String servicePort = (String) properties.get(HAZELCAST_SERVICE_PORT);
        if (servicePort != null) {
            port = Integer.parseInt(servicePort);
        }
    }
    return port;
}

此方法检查kubernetes客户端返回的每个端点返回的其他属性，以获取hazelcast端口配置.如果不存在，它将使用默认值5701.我猜这是需要配置的内容，但是，它一定不会影响其他群集，因此我可能必须使用自己的一些逻辑来扩展该策略.

This method checks the additional properties that get returned for each endpoint returned by the kubernetes client, for a hazelcast port configuration. If none exists, it uses the default 5701. I am guessing this is what needs to be configured, however, it must not impact the other cluster, so I may have to extend the strategy with some of my own logic.

推荐答案

可以将多个Hazelcast实例嵌入到单个POD中部署的应用程序中.然后，您可以控制集群的形成方式.它需要其他配置，但是您不必修改HazelcastKubernetesDiscoveryStrategy.

It's possible to embed multiple Hazelcast instances into an application deployed in a single POD. Then, you can control how the cluster are formed. It requires an additional configuration, but you don't have to modify HazelcastKubernetesDiscoveryStrategy.

我创建了一个示例应用程序来演示其工作方式.请在此处检查: https://github.com/leszko/hazelcast-code-samples/tree/kubernetes-embedded-multiple/hazelcast-integration/kubernetes/samples/embedded .

I've created a sample application to present how it works. Please check it here: https://github.com/leszko/hazelcast-code-samples/tree/kubernetes-embedded-multiple/hazelcast-integration/kubernetes/samples/embedded.

您的应用程序中有两个Hazelcast实例，因此您需要指定它们使用的端口.另外，使用 hazelcast-kubernetes 插件的参数，您可以配置构成聚在一起.

You have two Hazelcast instances in your application, so you need to specify the ports they use. Also, with the parameters of the hazelcast-kubernetes plugin, you can configure which Hazelcast instances form the cluster together.

例如，假设第一个Hazelcast实例应与当前Kubernetes命名空间中的所有其他Hazelcast实例组成一个群集，则其配置如下所示.

For example, assuming the first Hazelcast instance should form a cluster with all other Hazelcast instances in the current Kubernetes namespace, its configuration can look as follows.

Config config = new Config();
config.getNetworkConfig().setPort(5701);
config.getProperties().setProperty("hazelcast.discovery.enabled", "true");
JoinConfig joinConfig = config.getNetworkConfig().getJoin();
joinConfig.getMulticastConfig().setEnabled(false);

HazelcastKubernetesDiscoveryStrategyFactory discoveryStrategyFactory = new HazelcastKubernetesDiscoveryStrategyFactory();
Map<String, Comparable> properties = new HashMap<>();
joinConfig.getDiscoveryConfig().addDiscoveryStrategyConfig(new DiscoveryStrategyConfig(discoveryStrategyFactory, properties));

然后，第二个Hazelcast实例只能与同一应用程序一起形成集群.我们可以通过给它一个service-name参数以及环境变量的值来进行分离.

Then, the second Hazelcast instance could form the cluster only with the same application. We could make this separation by giving it a service-name parameter with the value from environment variables.

Config config = new Config();
config.getNetworkConfig().setPort(5702);
config.getProperties().setProperty("hazelcast.discovery.enabled", "true");
JoinConfig joinConfig = config.getNetworkConfig().getJoin();
joinConfig.getMulticastConfig().setEnabled(false);

HazelcastKubernetesDiscoveryStrategyFactory discoveryStrategyFactory = new HazelcastKubernetesDiscoveryStrategyFactory();
Map<String, Comparable> properties = new HashMap<>();
String serviceName = System.getenv("KUBERNETES_SERVICE_NAME");
properties.put("service-name", serviceName);
properties.put("service-port", "5702");
joinConfig.getDiscoveryConfig().addDiscoveryStrategyConfig(new DiscoveryStrategyConfig(discoveryStrategyFactory, properties));

GroupConfig groupConfig = new GroupConfig("separate");

Kubernetes模板

然后，在Kubernetes部署模板中，您需要配置两个端口:5701和5702.

ports:
- containerPort: 5701
- containerPort: 5702

以及带有服务名称的环境变量:

And the environment variable with the service name:

env:
- name: KUBERNETES_SERVICE_NAME
  value: hazelcast-separate-1

此外，您需要为每个Hazelcast实例创建两个服务.

Plus, you need to create two services, for each Hazelcast instance.

kind: Service
metadata:
  name: hazelcast-shared-1
spec:
  type: ClusterIP
  selector:
    app: hazelcast-app
  ports:
  - name: hazelcast-shared
    port: 5701

kind: Service
metadata:
  name: hazelcast-separate-1
spec:
  type: ClusterIP
  selector:
    app: hazelcast-app
  ports:
  - name: hazelcast-separate
  port: 5702

很显然，您可以用相同的方式使用服务标签而不是服务名称来分隔Hazelcast群集.

Obviously, in the same manner, you can separate Hazelcast clusters using service labels instead of service names.

这篇关于kubernetes中的两个单独的hazelcast集群的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！