在不同主机上运行的Kubernets POD无法建立TCP连接 [英] Kubernets PODs running on different host, not able to establish TCP connection

查看：62
本文介绍了在不同主机上运行的Kubernets POD无法建立TCP连接的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我有Kubernets 1.20.1集群，其中单个主机和单个工作器配置为 ipvs 模式.使用calico CNI calico/cni:v3.16.1 .在OS RHEL 8内核 4.18.0-240.10 上运行的集群，已禁用防火墙和selinux.

I have Kubernets 1.20.1 cluster with single master and single worker configured with ipvs mode. Using calico CNI calico/cni:v3.16.1. Cluster running on OS RHEL 8 kernel 4.18.0-240.10 with firewalld and selinux disabled.

在主服务器上运行一个 netshoot 吊舱( 10.1.30.130 )，在工作节点上运行另一个吊舱( 10.3.65.132 ).

Running one netshoot pod (10.1.30.130) on master and another pod (10.3.65.132) in worker node.

  1. 我可以在两个方向上ping通两个豆荚
  2. 如果在Web服务器模式下运行nc命令，则连接无法正常工作.我尝试在两台服务器上都运行Nginx，但无法从另一台服务器上获取一台服务器的HTTP流量.

在两个服务器上都运行tcpdump tcpdump -vv -nn -XX -i任何主机< PODIP> 我可以看到ping流量到达两个节点​​，但是TCP流量没有到达另一个节点.

Ran the tcpdump on both servers tcpdump -vv -nn -XX -i any host <PODIP> I can see ping traffic going to both nodes, but TCP traffic not reaching the other node.

iptables -vL |grep DROP 命令在两个节点上均未显示任何数据包丢弃.

iptables -vL | grep DROP command not showing any packet drop on both nodes.

我不知道TCP流量在哪里丢失，需要一些技巧来解决此问题.

I don't know where the TCP traffic getting lost, need some tips to troubleshoot this issue.

主节点iptables-save命令输出 

# Generated by iptables-save v1.8.4 on Sat Jan 16 18:52:50 2021
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-LOAD-BALANCER - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
-A KUBE-SERVICES ! -s 10.0.0.0/14 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
-A KUBE-FIREWALL -j KUBE-MARK-DROP
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
COMMIT
# Completed on Sat Jan 16 18:52:50 2021
# Generated by iptables-save v1.8.4 on Sat Jan 16 18:52:50 2021
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-FORWARD - [0:0]
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Sat Jan 16 18:52:50 2021
# Generated by iptables-save v1.8.4 on Sat Jan 16 18:52:50 2021
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:KUBE-KUBELET-CANARY - [0:0]
COMMIT
# Completed on Sat Jan 16 18:52:50 2021

Worker iptables-保存输出 

# Generated by iptables-save v1.8.4 on Sat Jan 16 18:53:58 2021
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-LOAD-BALANCER - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
-A KUBE-SERVICES ! -s 10.0.0.0/14 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
-A KUBE-FIREWALL -j KUBE-MARK-DROP
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
COMMIT
# Completed on Sat Jan 16 18:53:58 2021
# Generated by iptables-save v1.8.4 on Sat Jan 16 18:53:58 2021
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-FORWARD - [0:0]
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Sat Jan 16 18:53:58 2021
# Generated by iptables-save v1.8.4 on Sat Jan 16 18:53:58 2021
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:KUBE-KUBELET-CANARY - [0:0]
COMMIT
# Completed on Sat Jan 16 18:53:58 2021

推荐答案

我能够通过在VMware VM的 ens192 界面上的以下命令运行来解决此问题.

I was able to resolve this issue by running below command on ens192 interface on VMware VM on.

# cat /etc/sysconfig/network-scripts/ifcfg-ens192 | grep ETHTOOL
ETHTOOL_OPTS="-K ens192 tx-udp_tnl-csum-segmentation off; -K ens192 tx-udp_tnl-segmentation off"

从这里获得了提示: https://github.com/kubernetes-sigs/kubespray/issues/7268谢谢

got the tips from here: https://github.com/kubernetes-sigs/kubespray/issues/7268 Thanks

SR

这篇关于在不同主机上运行的Kubernets POD无法建立TCP连接的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
其他开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆