无法通过haproxy连接到cassandra容器 [英] Can't connect to cassandra container via haproxy

问题描述

我正在尝试将外部应用程序连接到在mesos集群上运行dockerized的Cassandra。

这些是我在mesos上运行的应用程序：

 集装箱ID图像命令创建状态端口名称
 137760ce852a cassandra：最新/docker-entrypoint.s15分钟前15分钟7000-7001 / tcp，7199 / tcp，9160 / tcp，0.0.0.0:31634- > 9042 / tcp mesos-1b65f33a-3d36-4bf4-8a77-32077d8d234a-S1.0db174cc-2e0c-4790-9cd7-1f142d08c6e2 
 fec5fc93ccfd cassandra：latest/docker-entrypoint.s22分钟前向上22分钟7000-7001 / tcp，7199 / tcp，9160 / tcp，0.0.0.0:31551->9042/tcp mesos-1b65f33a-3d36-4bf4-8a77-32077d8d234a-S1.0022a3d2-d695-43c4-b22f-f5274cbd03ce 
 ca729ee628bb tobilg / mesos-dns./bootstrap.sh大约一小时前Up大约一小时mesos-1b65f33a-3d36-4bf4-8a77-32077d8d234a-S1.12593777-2295-42fa-a56d-1d3cc9fc70ff 
 3921002a8a5b python：3/ bin / sh -c'env> env大约一小时前Up大约一小时0.0.0.0:31295->8080/tcp mesos-1b65f33a-3d36-4bf4-8a77-32077d8d234a-S1。 b101ab59-2538-416f-80cf-29215794bd37 
  

名为 peek 的应用程式只是用于测试建议。我可以在以下网址访问它： http://192.168.56.101:10001 ，没有问题。

两个cassandra实例是种子，另一个用于放大;

在marathon上部署cassandra应用程序的json描述如下：

/ cassandra-seed

  {
id：cassandra-seed b $ bconstraints：[[hostname，CLUSTER，docker-sl-vm]]，
container：{
type：DOCKER，
docker：{
image：cassandra：latest，
network：BRIDGE，
portMappings：[{containerPort：9042， hostPort：0，servicePort：0，protocol：tcp}] 
} 
}，
cpus：0.5，
mem：512.0 ，
instances：1，
backoffSeconds：1，
backoffFactor：1.15，
maxLaunchDelaySeconds：3600 
} / code> 

/ cassandra

  {
id：cassandra，
constraints：[[hostname，CLUSTER，docker-sl-vm]] $ bcontainer：{
type：DOCKER，
docker：{
image：cassandra：latest，
network BRIDGE，
portMappings：[{containerPort：9042，hostPort：0，servicePort：0，protocol：tcp}] 
} 
 }，
env：{
CASSANDRA_SEED_COUNT：1，
CASSANDRA_SEEDS：cassandra-seed.marathon.mesos
}，
 cpus：0.5，
mem：512.0，
instances：1，
backoffSeconds：1，
backoffFactor：1.15，
 maxLaunchDelaySeconds：3600 
} 
  

haproxy配置如下：

 全局
守护程序
日志127.0.0.1 local0 
日志127.0.0.1 local1 notice 
 maxconn 4096 
 tune.ssl.default-dh-param 2048 
 
默认值
日志全局
重试3 
 maxconn 2000 
 timeout连接5s 
 timeout client 50s 
 timeout server 50s 
 
 listen stats 
 bind 127.0.0.1:9090 
 balance 
 mode http 
 stats enable 
 stats auth admin：admin 
 
 frontend marathon_http_in 
 bind *：80 
 mode http 
 
 frontend marathon_http_appid_in 
 bind *：81 
 mode http 
 
 frontend marathon_https_in 
 bind *：443 ssl crt /etc/ssl/xip.io/xip.io.pem 
 mode http 
 
 frontend cassandra_10003 
 bind *：10003 
 mode tcp 
 use_backend cassandra_10003 
 
 frontend cassandra-seed_10002 
 bind *：10002 
模式tcp 
 use_backend cassandra-seed_10002 
 
 frontend dns_10000 
 bind *：10000 
 mode tcp 
 use_backend dns_10000 
 
 frontend peek_10001 
 bind *：10001 
 mode tcp 
 use_backend peek_10001 
 
后端cassandra_10003 
 balance roundrobin 
模式tcp 
服务器docker-sl -vm_31634 192.168.56.102:31634 
 
后端cassandra-seed_10002 
 balance roundrobin 
模式tcp 
服务器docker-sl-vm_31551 192.168.56.102:31551 
 
 backend dns_10000 
 balance roundrobin 
模式tcp 
服务器docker-sl-vm_31314 192.168.56.102:31314 
 
后端peek_10001 
 balance roundrobin 
 mode tcp 
 server docker-sl-vm_31295 192.168.56.102:31295 
  

应用程序我试图连接到Cassandra是一个Play应用程序。我的设置如下：

  akka.persistence {
 journal.plugin =cassandra-journal
 snapshot-store.plugin =cassandra-snapshot-store
} 
 
 cassandra-journal.contact-points = [192.168.56.101:10003] 
 cassandra -snapshot-store.contact-points = [192.168.56.101:10003] 
  

但是当我尝试访问它，我得到以下错误：

 ！ @ 6o380dcg9  - 内部服务器错误，（GET）[/ issues / list]  - > 
 
 play.api.Application $$ anon $ 1：执行异常[[TimeoutException：deadline passed]] 
 at play.api.Application $ class.handleError（Application.scala：296）〜 [play_2.11-2.3.10.jar：2.3.10] 
 at play.api.DefaultApplication.handleError（Application.scala：402）[play_2.11-2.3.10.jar：2.3.10] 
 at play.core.server.netty.PlayDefaultUpstreamHandler $$ anonfun $ 14 $$ anonfun $ apply $ 1.applyOrElse（PlayDefaultUpstreamHandler.scala：205）[play_2.11-2.3.10.jar：2.3.10] 
 at play.core.server.netty.PlayDefaultUpstreamHandler $$ anonfun $ 14 $$ anonfun $ apply $ 1.applyOrElse（PlayDefaultUpstreamHandler.scala：202）[play_2.11-2.3.10.jar：2.3.10] 
 at scala.runtime.AbstractPartialFunction.apply（AbstractPartialFunction.scala：36）[scala-library-2.11.7.jar：na] 
导致：java.util.concurrent.TimeoutException：deadline passed a 
 at akka .actor.dsl.Inbox $ InboxActor $$ anonfun $ receive $ 1.applyOrElse（Inbox.scala：117）〜[akka-actor_2.11-2.4.0.jar：na] 
 at scala.PartialFunction $ AndThen。 applyOrElse（PartialFunction.scala：189）〜[scala-library-2.11.7.jar：na] 
 at akka.actor.Actor $ class.aroundReceive（Actor.scala：480）〜[akka-actor_2.11 -2.4.0.jar：na] 
 at akka.actor.dsl.Inbox $ InboxActor.aroundReceive（Inbox.scala：62）〜[akka-actor_2.11-2.4.0.jar：na] 
 at akka.actor.ActorCell.receiveMessage（ActorCell.scala：525）〜[akka-actor_2.11-2.4.0.jar：na] 
 [错误] cddcSession  - 创建池到/172.17时出错。 0.2：9042 
 com.datastax.driver.core.TransportException：[/172.17.0.2:9042]无法在com.datastax.driver.core.Connection连接
< init>（Connection.java ：109）〜[cassandra-driver-core-2.1.5.jar：na] 
 at com.datastax.driver.core.PooledConnection。< init>（PooledConnection.java:32）〜[cassandra- -core-2.1.5.jar：na] 
 at com.datastax.driver.core.Connection $ Factory.open（Connection.java:586）〜[cassandra-driver-core-2.1.5.jar： na] 
 at com.datastax.driver.core.SingleConnectionPool。< init>（SingleConnectionPool.java:76）〜[cassandra-driver-core-2.1.5.jar：na] 
 at com .datastax.driver.core.HostConnectionPool.newInstance（HostConnectionPool.java:35）〜[cassandra-driver-core-2.1.5.jar：na] 
导致：org.jboss.netty.channel.ConnectTimeoutException：连接超时：/172.17.0.2:9042 
在org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout（NioClientBoss.java:139）〜[netty-3.9.9.Final.jar：na ] 
 at org.jboss.netty.channel.socket.nio.NioClientBoss.process（NioClientBoss.java:83）〜[netty-3.9.9.Final.jar：na] 
 at org.jboss .netty.channel.socket.nio.AbstractNioSelector.run（AbstractNioSelector.java:337）〜[netty-3.9.9.Final.jar：na] 
 at org.jboss.netty.channel.socket.nio。 NioClientBoss.run（NioClientBoss.java:42）〜[netty-3.9.9.Final.jar：na] 
 at org.jboss.netty.util.ThreadRenamingRunnable.run（ThreadRenamingRunnable.java:108）〜[netty -3.9.9.Final.jar：na] 
 [错误] cddcSession  - 创建池到/172.17.0.2:9042时出错
  

有谁知道如何解决这个问题？

更新=== ========================

有趣的是，创建了我的应用程序的键空间（akka，akka_snapshots）：

  cqlsh&描述关键空间; 
 akka_snapshot system_auth system system_distributed system_traces akka 
  

UPDATE 2 ====== =====================

我刚刚注意到，甚至连接应用程序直接运行cassandra（不通过haproxy）。所以，我改变了portMapping为：

 portMappings：[{containerPort：9042，hostPort： 0，servicePort：9042，protocol：tcp}] 
  

HOWEVER ，它只允许我启动一台机器，因为servicePort声明。

问题是进入端口映射。任何线索？

解决方案

我知道您正在使用 haproxy 服务发现 Cassandra 集群。如果是这样，如果您没有更改马拉松的任务更改配置（缩放等）的机制，它将不会成功。

你的 Cassandra 节点无法相互通信的问题可能是因为 / cassandra 应用程序没有引用 / cassandra-seed 。

根据 Cassandra Docker图片docs a>您应该能够动态配置 CASSANDRA_SEEDS env参数。

为了能够使用服务名称 cassandra-seed.marathon.mesos 如果有必要将其解析为IP地址首先IMHO：

 CASSANDRA_SEEDS：$（host cassandra-seed.marathon.mesos | awk'/ has address / {print $ 4}'）
  pre> 
 
 
会理论上工作（例如，如果您的应用只有一个实例）。
 
 
 
因为你似乎使用Mesos DNS，可能会有一个问题，因为目前（v0.4.0）只有内部IP地址广告（见 Issue ）。您可能必须回到一个真正的Mesos DNS客户端，它可以解析SRV记录，以正确地映射到Mesos Slave IP地址和端口。 
 
 
 
或者，您可以自行解析 dig 结果，并将其用作 CASSANDRA_SEEDS  env参数：
  dig _cassandra-seed._tcp.marathon.mesos SRV 
  
请参阅 Mesos DNS文档。
 
 
 
  mesosdns-cli 可以处理这个，但是需要在Docker容器中使用Node.js运行时。因此，您必须创建您自己的 cassandra  Docker镜像的派生物。
 
I am trying to connect an external app to Cassandra which is running dockerized on a mesos cluster. 

These are the the apps I have running on mesos:
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                        NAMES
137760ce852a        cassandra:latest    "/docker-entrypoint.s"   15 minutes ago      Up 15 minutes       7000-7001/tcp, 7199/tcp, 9160/tcp, 0.0.0.0:31634->9042/tcp   mesos-1b65f33a-3d36-4bf4-8a77-32077d8d234a-S1.0db174cc-2e0c-4790-9cd7-1f142d08c6e2
fec5fc93ccfd        cassandra:latest    "/docker-entrypoint.s"   22 minutes ago      Up 22 minutes       7000-7001/tcp, 7199/tcp, 9160/tcp, 0.0.0.0:31551->9042/tcp   mesos-1b65f33a-3d36-4bf4-8a77-32077d8d234a-S1.0022a3d2-d695-43c4-b22f-f5274cbd03ce
ca729ee628bb        tobilg/mesos-dns    "./bootstrap.sh"         About an hour ago   Up About an hour                                                                 mesos-1b65f33a-3d36-4bf4-8a77-32077d8d234a-S1.12593777-2295-42fa-a56d-1d3cc9fc70ff
3921002a8a5b        python:3            "/bin/sh -c 'env >env"   About an hour ago   Up About an hour    0.0.0.0:31295->8080/tcp                                      mesos-1b65f33a-3d36-4bf4-8a77-32077d8d234a-S1.b101ab59-2538-416f-80cf-29215794bd37
the app called peek is just being used for testing proposals. I can access it at the URL: http://192.168.56.101:10001 with no problems.

The 2 cassandra instances are a seed and another one for scaling up; forming a cluster.

The json description for deployment of the cassandra applications on marathon are as following:

/cassandra-seed
{
    "id": "cassandra-seed",
    "constraints": [["hostname", "CLUSTER", "docker-sl-vm"]],
    "container": {
        "type": "DOCKER",
        "docker": {
            "image": "cassandra:latest",
            "network": "BRIDGE",
            "portMappings": [ {"containerPort": 9042,"hostPort": 0,"servicePort": 0,"protocol": "tcp"} ]
        }
    },
    "cpus": 0.5,
    "mem": 512.0,
    "instances": 1,
    "backoffSeconds": 1,
    "backoffFactor": 1.15,
    "maxLaunchDelaySeconds": 3600
}
/cassandra
{
    "id": "cassandra",
    "constraints": [["hostname", "CLUSTER", "docker-sl-vm"]],
    "container": {
        "type": "DOCKER",
        "docker": {
            "image": "cassandra:latest",
            "network": "BRIDGE",
            "portMappings": [ {"containerPort": 9042,"hostPort": 0,"servicePort": 0,"protocol": "tcp"} ]
        }
    },
    "env": {
            "CASSANDRA_SEED_COUNT": "1",
        "CASSANDRA_SEEDS": "cassandra-seed.marathon.mesos"
    },
    "cpus": 0.5,
    "mem": 512.0,
    "instances": 1,
    "backoffSeconds": 1,
    "backoffFactor": 1.15,
    "maxLaunchDelaySeconds": 3600
}
haproxy configuration is as following:
global
  daemon
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  maxconn 4096
  tune.ssl.default-dh-param 2048

defaults
  log               global
  retries           3
  maxconn           2000
  timeout connect   5s
  timeout client    50s
  timeout server    50s

listen stats
  bind 127.0.0.1:9090
  balance
  mode http
  stats enable
  stats auth admin:admin

frontend marathon_http_in
  bind *:80
  mode http

frontend marathon_http_appid_in
  bind *:81
  mode http

frontend marathon_https_in
  bind *:443 ssl crt /etc/ssl/xip.io/xip.io.pem
  mode http

frontend cassandra_10003
  bind *:10003
  mode tcp
  use_backend cassandra_10003

frontend cassandra-seed_10002
  bind *:10002
  mode tcp
  use_backend cassandra-seed_10002

frontend dns_10000
  bind *:10000
  mode tcp
  use_backend dns_10000

frontend peek_10001
  bind *:10001
  mode tcp
  use_backend peek_10001

backend cassandra_10003
  balance roundrobin
  mode tcp
  server docker-sl-vm_31634 192.168.56.102:31634

backend cassandra-seed_10002
  balance roundrobin
  mode tcp
  server docker-sl-vm_31551 192.168.56.102:31551

backend dns_10000
  balance roundrobin
  mode tcp
  server docker-sl-vm_31314 192.168.56.102:31314

backend peek_10001
  balance roundrobin
  mode tcp
  server docker-sl-vm_31295 192.168.56.102:31295
The application I am trying to connect to Cassandra is a Play application. I am setting it like this:
akka.persistence {
  journal.plugin = "cassandra-journal"
  snapshot-store.plugin = "cassandra-snapshot-store"
}

cassandra-journal.contact-points = ["192.168.56.101:10003"]
cassandra-snapshot-store.contact-points = ["192.168.56.101:10003"]
The app starts up OK, but when I try to access it, I get the following error:
! @6o380dcg9 - Internal server error, for (GET) [/issues/list] ->

play.api.Application$$anon$1: Execution exception[[TimeoutException: deadline passed]]
        at play.api.Application$class.handleError(Application.scala:296) ~[play_2.11-2.3.10.jar:2.3.10]
        at play.api.DefaultApplication.handleError(Application.scala:402) [play_2.11-2.3.10.jar:2.3.10]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$14$$anonfun$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:205) [play_2.11-2.3.10.jar:2.3.10]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$14$$anonfun$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:202) [play_2.11-2.3.10.jar:2.3.10]
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) [scala-library-2.11.7.jar:na]
Caused by: java.util.concurrent.TimeoutException: deadline passed
        at akka.actor.dsl.Inbox$InboxActor$$anonfun$receive$1.applyOrElse(Inbox.scala:117) ~[akka-actor_2.11-2.4.0.jar:na]
        at scala.PartialFunction$AndThen.applyOrElse(PartialFunction.scala:189) ~[scala-library-2.11.7.jar:na]
        at akka.actor.Actor$class.aroundReceive(Actor.scala:480) ~[akka-actor_2.11-2.4.0.jar:na]
        at akka.actor.dsl.Inbox$InboxActor.aroundReceive(Inbox.scala:62) ~[akka-actor_2.11-2.4.0.jar:na]
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:525) ~[akka-actor_2.11-2.4.0.jar:na]
[error] c.d.d.c.Session - Error creating pool to /172.17.0.2:9042
com.datastax.driver.core.TransportException: [/172.17.0.2:9042] Cannot connect
        at com.datastax.driver.core.Connection.<init>(Connection.java:109) ~[cassandra-driver-core-2.1.5.jar:na]
        at com.datastax.driver.core.PooledConnection.<init>(PooledConnection.java:32) ~[cassandra-driver-core-2.1.5.jar:na]
        at com.datastax.driver.core.Connection$Factory.open(Connection.java:586) ~[cassandra-driver-core-2.1.5.jar:na]
        at com.datastax.driver.core.SingleConnectionPool.<init>(SingleConnectionPool.java:76) ~[cassandra-driver-core-2.1.5.jar:na]
        at com.datastax.driver.core.HostConnectionPool.newInstance(HostConnectionPool.java:35) ~[cassandra-driver-core-2.1.5.jar:na]
Caused by: org.jboss.netty.channel.ConnectTimeoutException: connection timed out: /172.17.0.2:9042
        at org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:139) ~[netty-3.9.9.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83) ~[netty-3.9.9.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[netty-3.9.9.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) ~[netty-3.9.9.Final.jar:na]
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.9.9.Final.jar:na]
[error] c.d.d.c.Session - Error creating pool to /172.17.0.2:9042
Does anyone know how to fix this? What am I doing wrong?

Thank you in advance...

UPDATE =============================

Interesting thing is that the keyspaces for my application were created (akka, akka_snapshots):
cqlsh> describe keyspaces;
akka_snapshot  system_auth  system  system_distributed  system_traces  akka
UPDATE 2 =============================

I have just noticed that I can't even connect the app directly to the running cassandra (without going through the haproxy). So, I've changed the portMapping to:
"portMappings": [ {"containerPort": 9042,"hostPort": 0,"servicePort": 9042,"protocol": "tcp"} ]
and it worked. HOWEVER, it only allow me to startup one machine, because of the servicePort declaration.

The problem is right into the port mapping. Any clue?
解决方案
I  understand you're using haproxy for the service discovery of the Cassandra cluster. If so, it won't be successful if you don't have a mechanism that updates the configuration once the tasks from Marathon are changed (scaling etc.).

The problem why your Cassandra node can't talk to each other is presumably that the /cassandra app has no reference to /cassandra-seed . 

According to the Cassandra Docker image docs you should be able to configure the CASSANDRA_SEEDS env parameter dynmically.

To be able to use the service name cassandra-seed.marathon.mesos if would be necessary to resolve it to an IP address first IMHO:
"CASSANDRA_SEEDS": "$(host cassandra-seed.marathon.mesos | awk '/has address/ { print $4 }')"
would theoretically work (e.g. if your app has just one instance).

As you seem to use Mesos DNS, there can be a problem because currently (v0.4.0) only internal IP addresses are advertised (see Issue). You might have to fall back to a "real" Mesos DNS client which can resolve SRV records to correctly map those to Mesos Slave IP adresses and ports. 

Or, you can parse the dig results yourself and use this as an input for the CASSANDRA_SEEDS env parameter:
dig _cassandra-seed._tcp.marathon.mesos SRV
see Mesos DNS docs.

mesosdns-cli can handle this, but requires a Node.js runtime in the Docker container where it should be used. You'd therefore have to create your own derivate of the cassandra Docker image.

                        
这篇关于无法通过haproxy连接到cassandra容器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！