kubernetes配置入口，ingress_nginx服务外部IP待处理 [英] kubernetes configuration Ingress，ingress_nginx service external IP pending
问题描述
我在win10机器上有一个虚拟机centos(版本7.4)，我不使用AWS，Google云服务或Azure.我将主服务器和节点放在一台机器上.我最初的问题域有5个组件，我将它们配置为ClusterIP，因此它们可以彼此通信(eureka，config，api，uaa，zipkin).现在，我只需要在外面进行api对话.但是总之，为了方便起见，我制作了两个组件(api和eureka).但是现在，api需要从集群外部接收.这样我就可以配置入口.进入时，我需要配置rbac.我将我的Yaml文件放在此处，并显示错误消息.
I have a virtual machine centos(ver 7.4) on win10 machine, I do not use AWS, Google cloud service, nor Azure. I put master and node in one machine. My original problem domain have 5 components, I configure them as ClusterIP, so they could communicate with each other(eureka, config,api,uaa,zipkin). Now I only need api talk outside. But for short, I make two components for convenience (api and eureka). But now, api needs to receive from outside of cluster. So that I configure ingress. When ingress, I need to configure rbac. I put my yaml file here with error message.
eureka_pod.yaml
eureka_pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: gearbox-rack-eureka-server
labels:
app: gearbox-rack-eureka-server
purpose: platform_eureka_demo
spec:
containers:
- name: gearbox-rack-eureka-server
image: 192.168.1.229:5000/gearboxrack/gearbox-rack-eureka-server
ports:
- containerPort: 8761
eureka_svc.yaml
eureka_svc.yaml
apiVersion: v1
kind: Service
metadata:
name: gearbox-rack-eureka-server
labels:
name: gearbox_rack_eureka_server
spec:
selector:
app: gearbox-rack-eureka-server
type: ClusterIP
ports:
- port: 8761
name: tcp
api_pod.yaml:
api_pod.yaml:
apiVersion: v1
kind: Pod
metadata:
name: gearbox-rack-api-gateway
labels:
app: gearbox-rack-api-gateway
purpose: platform-demo
spec:
containers:
- name: gearbox-rack-api-gateway
image: 192.168.1.229:5000/gearboxrack/gearbox-rack-api-gateway
ports:
- containerPort: 5555
env:
- name: EUREKA_SERVER
value: http://gearbox-rack-eureka-server:8761
- name: CONFIG_SERVER
value: http://gearbox-rack-config-server:8888
- name: CONFIG_PROFILE
value: docker
- name: CONFIG_LABEL
value: master
- name: ZIPKIN_SERVER
value: http://gearbox-rack-zipkin-server:9411
api_svc.yaml:
api_svc.yaml:
apiVersion: v1
kind: Service
metadata:
name: gearbox-rack-api-gateway
labels:
name: gearbox-rack-api-gateway
spec:
selector:
app: gearbox-rack-api-gateway
type: ClusterIP
ports:
- port: 5555
name: tcp
ingress_nginx_role_rb.yaml:
ingress_nginx_role_rb.yaml:
apiVersion: v1
kind: ServiceAccount
metadata:
name: lb
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-normal
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-minimal
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
- "ingress-controller-leader-dev"
- "ingress-controller-leader-prod"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-minimal
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-minimal
subjects:
- kind: ServiceAccount
name: lb
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-normal
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-normal
subjects:
- kind: ServiceAccount
name: lb
namespace: kube-system
nginx_default_backend.yaml:
nginx_default_backend.yaml:
kind: Service
apiVersion: v1
metadata:
name: nginx-default-backend
namespace: kube-system
spec:
ports:
- port: 80
targetPort: http
selector:
app: nginx-default-backend
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: nginx-default-backend
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
app: nginx-default-backend
spec:
terminationGracePeriodSeconds: 60
containers:
- name: default-http-backend
image: chenliujin/defaultbackend
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
ports:
- name: http
containerPort: 8080
protocol: TCP
ingress_nginx_ctl.yaml:
ingress_nginx_ctl.yaml:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
spec:
type: NodePort
selector:
app: ingress-nginx
ports:
- name: http
port: 80
nodePort: 31080
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: ingress-nginx
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
app: ingress-nginx
spec:
terminationGracePeriodSeconds: 60
serviceAccount: lb
containers:
- image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.15.0
name: ingress-nginx
imagePullPolicy: Always
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/nginx-default-backend
ingress_nginx_res.yaml:
ingress_nginx_res.yaml:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
spec:
rules:
- host:
http:
paths:
- path: /eureka
backend:
serviceName: gearbox-rack-eureka-server
servicePort: 8761
- host:
http:
paths:
- path: /
backend:
serviceName: gearbox-rack-api-gateway
servicePort: 5555
当我尝试172.16.100.88:31080/uaa/login时(我的虚拟机当前IP是172.16.100.88)，它显示以下连接问题:
when I try 172.16.100.88:31080/uaa/login, (my virtual machine current IP is 172.16.100.88) it says following connection problme:
[centos@master ~]$ sudo curl http://172.16.100.88:31080/uaa/login
curl: (7) Failed connect to 172.16.100.88:31080; Connection refused
我检查了ingress-nginx pod，看来请求尚未到达nginx.
I check ingress-nginx pod, it seems request has not yet reached nginx.
[centos@master ~]$ sudo kubectl logs ingress-nginx-5c6d78668c-c78rd -n kube-system
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.15.0
Build: git-df61bd7
Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------
W0606 03:24:39.256706 7 client_config.go:533] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0606 03:24:39.257090 7 main.go:158] Creating API client for https://10.96.0.1:443
I0606 03:24:39.269897 7 main.go:202] Running in Kubernetes Cluster version v1.10 (v1.10.3) - git (clean) commit 2bba0127d85d5a46ab4b778548be28623b32d0b0 - platform linux/amd64
I0606 03:24:39.271426 7 main.go:84] validated kube-system/nginx-default-backend as the default backend
I0606 03:24:39.403474 7 stat_collector.go:77] starting new nginx stats collector for Ingress controller running in namespace (class nginx)
I0606 03:24:39.403505 7 stat_collector.go:78] collector extracting information from port 18080
I0606 03:24:39.415000 7 nginx.go:278] starting Ingress controller
I0606 03:24:40.518572 7 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"my-ingress", UID:"041fb77f-6939-11e8-971e-fa163e39f742", APIVersion:"extensions", ResourceVersion:"1040", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/my-ingress
I0606 03:24:40.615398 7 nginx.go:299] starting NGINX process...
I0606 03:24:40.615582 7 leaderelection.go:175] attempting to acquire leader lease kube-system/ingress-controller-leader-nginx...
I0606 03:24:40.618811 7 controller.go:168] backend reload required
I0606 03:24:40.618835 7 stat_collector.go:34] changing prometheus collector from to default
I0606 03:24:40.623350 7 leaderelection.go:184] successfully acquired lease kube-system/ingress-controller-leader-nginx
I0606 03:24:40.623400 7 status.go:196] new leader elected: ingress-nginx-5c6d78668c-c78rd
I0606 03:24:40.744438 7 controller.go:177] ingress backend successfully reloaded...
I0606 03:25:40.635267 7 status.go:361] updating Ingress default/my-ingress status to [{ }]
I0606 03:25:40.640390 7 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"my-ingress", UID:"041fb77f-6939-11e8-971e-fa163e39f742", APIVersion:"extensions", ResourceVersion:"1185", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/my-ingress
当我登录到pod gearbox-rack-api-gateway时，我可以清楚地看到它重定向到我期望的页面.所以我的yaml文件中肯定有一些配置错误.
When I login in to pod gearbox-rack-api-gateway, I could see clearly it redirects to the page I expected. so there must be some configuration wrong in my yaml files.
[root@master8g ingress]# kubectl exec -it gearbox-rack-api-gateway -- /bin/bash
root@gearbox-rack-api-gateway:/# curl http://localhost:5555/uaa/login
<!DOCTYPE html>
<!--[if IE 8]> <html lang="en" class="ie8 no-js"> <![endif]-->
<!--[if IE 9]> <html lang="en" class="ie9 no-js"> <![endif]-->
<!--[if !IE]><!-->
<html lang="en" class="no-js">
<!--<![endif]-->
<!-- BEGIN HEAD -->
<head>
....
</head>
<!-- END HEAD -->
<body>
<div class="blank"></div>
...
</div>
<script>
document.getElementById("username").focus();
</script>
</body>
</html>
================================================ ==================
=================================================================
在我的虚拟机中，我键入telnet localhost 31080，但被拒绝.但telnet -6 localhost 31080成功.并且netstat -anp | less找到31080绑定kube-proxy.我在开始脚本中放入了sysctl -q -w net.ipv6.conf.all.disable_ipv6=1和sysctl -w net.ipv6.conf.default.disable_ipv6=1，但是得到了相同的结果.
In my virtual machine, I type telnet localhost 31080, rejected. but telnet -6 localhost 31080 succeed. And netstat -anp | less find 31080 binding kube-proxy. I put sysctl -q -w net.ipv6.conf.all.disable_ipv6=1 and sysctl -w net.ipv6.conf.default.disable_ipv6=1 in my starting script, but got same result.
tcp6 0 0 :::31080 :::* LISTEN 7436/kube-proxy
================================================ ===============
==============================================================
昨天有关Ipv6的问题很愚蠢.我错误地配置了/etc/hosts.现在，我可以使用telnet localhost 31080了，但是当我卷曲时 http://localhost:31080/uaa/login ，它在那里悬挂了很长时间.所以Pod在听.当我发出命令curl http://localhost:31080/uaa/login 时， ，我检查了几个Pod的日志.日志没有显示错误，也没有日志说端口31080已发送请求.我检查了ingress-nginx荚日志:我在这里粘贴了一些.
Yesterday question about Ipv6 is stupid. I misconfigured /etc/hosts. Now I have telnet localhost 31080 work, but when I do curl http://localhost:31080/uaa/login, it hangs there for long time. So pod is listening. When I issue the command curl http://localhost:31080/uaa/login, at the same time, I check several pods' log. Log has shown no error and has no log to say the port 31080 has been sent request. I checked ingress-nginx pod logs: I paste some here.
I0608 07:15:31.436658 5 leaderelection.go:184] successfully acquired lease kube-system/ingress-controller-leader-nginx
I0608 07:15:31.436683 5 status.go:196] new leader elected: ingress-nginx-6ccc799fbc-tl588
I0608 07:15:31.493595 5 controller.go:177] ingress backend successfully reloaded...
I0608 07:16:31.441317 5 status.go:361] updating Ingress default/my-ingress status to [{ }]
I0608 07:16:31.444803 5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"my-ingress", UID:"b96b555d-6aeb-11e8-acff-080027709399", APIVersion:"extensions", ResourceVersion:"1270", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/my-ingress
[root@master8g ingress]# curl http://localhost:31080/uaa/login
^C
[root@master8g ingress]# telnet localhost 31080
Trying ::1...
Connected to localhost.
Escape character is '^]'.
^C^CTerminated
我自己的猜测是入口nginx pod的名称空间是kube-system，nginx服务的名称空间是默认，my-ingress的名称空间是默认. nginx-default-backend的名称空间是kube-system.是否禁止跨名称空间流量.
My own wild guess is that ingress nginx pod's namespace is kube-system, nginx service's name space is default, my-ingress's namespace is default. nginx-default-backend's name space is kube-system. Whether cross namespace traffic is forbidden.
专家您需要哪种日志?
Experts what kind of logs do you need?
================================================ ========
=======================================================
在定义所有入口控制器之后，将入口资源定义为默认名称空间，现在我确实走得更远了.现在，nginx重定向了我的http request to https:请求.如何禁用此功能将使我的入口完全正常工作.
After define all ingress controller, ingress resources as default namespace, now I did get move further. Now nginx redirect my http request to https: request. How to disable this feature will make my ingress working wholly.
[root@master8g ingress]# curl http://172.16.100.88:31080/uaa/login
<html>
<head><title>308 Permanent Redirect</title></head>
<body bgcolor="white">
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.13.12</center>
</body>
</html>
我注意到ingress-nginx收到了请求，如下所示:
I notice ingress-nginx receive the request, shown as below:
[root@master8g ingress]# kubectl logs ingress-nginx-6ccc799fbc-7clb6
172.16.100.88 - [172.16.100.88] - - [08/Jun/2018:09:16:01 +0000] "GET / HTTP/1.1" 308 188 "-" "curl/7.29.0" 83 0.000 [default-gearbox-rack-api-gateway-5555] - - - - 322eb1a195d940d5aa690d81ba9d0a4b
172.16.100.88 - [172.16.100.88] - - [08/Jun/2018:09:16:15 +0000] "GET /uaa/login HTTP/1.1" 308 188 "-" "curl/7.29.0" 92 0.000 [default-gearbox-rack-api-gateway-5555] - - - - 31dabba6b0e6cbbc899ec1248ffacd92
172.16.100.88 - [172.16.100.88] - - [08/Jun/2018:09:20:39 +0000] "GET / HTTP/1.1" 308 188 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36" 383 0.000 [default-gearbox-rack-api-gateway-5555] - - - - a00e1c447b581e46566fbb216134e993
172.16.100.88 - [172.16.100.88] - - [08/Jun/2018:09:20:39 +0000] "\x16\x03\x01\x02\x00\x01\x00\x01\xFC\x03\x03\x1A\xBB\xB6R\xB3Y\xCF;\x95\xBD[B\xC4\x09\x15\xE8\xE35C6\x82\xB0mN\xE9~p\xC4\x87M\x89H O\xCC4\xB6'\xCC\xAA\xA9\xC5\xB6\xDC\xFE" 400 174 "-" "-" 0 0.001 [] - - - - c914c5f584a1cec087e33a5ebd8b1c8d
172.16.100.88 - [172.16.100.88] - - [08/Jun/2018:09:20:39 +0000] "\x16\x03\x01\x02\x00\x01\x00\x01\xFC\x03\x03\xD15\xD8&\x9F\x0E\xBB\xFF\xFE\xF0\xD5x\xD17\xF6\x01\xD1\x91\x0E\x1E\x84\x8C\x94(\xBD\xF81\xA0\x80\xB4\xE7q \xB5\x1D\xEEG9\x09g\xBCi\x85BC\xCB\x9Ag\xD9@'\xB6 \x18\x8F\xC8" 400 174 "-" "-" 0 0.002 [] - - - - 8164c246d67cb97331ec5230679e7e6f
172.16.100.88 - [172.16.100.88] - - [08/Jun/2018:09:20:39 +0000] "\x16\x03\x01\x00\xAE\x01\x00\x00\xAA\x03\x03U\xC5\x22.\x9Eq\xCD\xDFi\xE0\x0B\x11\x93\xD0L\x9C\xCE\xBA\xD1\xA3\xE6\x124y\xA6\xEA\xB8\x88I\x97\x1F\xB8\x00\x00\x1C\x8A\x8A\xC0+\xC0/\xC0,\xC00\xCC\xA9\xCC\xA8\xC0\x13\xC0\x14\x00\x9C\x00\x9D\x00/\x005\x00" 400 174 "-" "-" 0 0.001 [] - - - - 0556f985d88e3ede2217459ab75ab4f4
ingress_nginx_res.yaml:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
spec:
rules:
- host:
http:
paths:
- path: /
backend:
serviceName: gearbox-rack-api-gateway
servicePort: 5555
推荐答案
apiVersion: v1
kind: Service
metadata:
name: gearbox-rack-eureka-server
labels:
name: gearbox_rack_eureka_server
spec:
selector:
app: gearbox-rack-eureka-server
type: ClusterIP
ports:
- port: 8761
name: tcp
名称:tcp->协议:TCP?
name: tcp -> protocol: TCP ?
这篇关于kubernetes配置入口，ingress_nginx服务外部IP待处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
