Kubernetes Ingress-nginx保留源IP [英] Kubernetes ingress-nginx preserve source IP

查看：177
本文介绍了Kubernetes Ingress-nginx保留源IP的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我有一个位于群集前面的虚拟机.当前，它正在运行HAProxy(带有use-proxy-protocol: "true").我的最终目标是允许与默认后端关联的Pod能够读取实际的源客户端源IP.

I have a vm that sits in front of the cluster. Currently it is running HAProxy (with use-proxy-protocol: "true"). My end goal is to allow the pods associated with the default backend to be able to read the actual source client source IP.

以下是打开了use-proxy-protocol的示例日志:

Here's a sample log of with use-proxy-protocol turned on:

10.244.0.0 - [10.244.0.0] - - [10/Jan/2018:23:06:42 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 367 0.002 [upstream-default-backend] 10.244.3.101:80 16 0.002 200
10.244.0.0 - [10.244.0.0] - - [10/Jan/2018:23:06:59 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "curl/7.54.0" 91 0.074 [upstream-default-backend] 10.244.3.101:80 16 0.074 200
10.244.0.0 - [10.244.0.0] - - [10/Jan/2018:23:09:51 +0000] "PROXY TCP4 127.0.0.1 127.0.0.1 43088 80" 400 173 "-" "-" 0 0.001 [] - - - -
10.244.0.0 - [10.244.0.0] - - [10/Jan/2018:23:09:59 +0000] "PROXY TCP4 127.0.0.1 127.0.0.1 43092 80" 400 173 "-" "-" 0 0.001 [] - - - -
10.244.0.0 - [10.244.0.0] - - [10/Jan/2018:23:10:09 +0000] "PROXY TCP4 127.0.0.1 127.0.0.1 43096 80" 400 173 "-" "-" 0 0.002 [] - - - -
I0110 23:11:42.050971       5 controller.go:211] backend reload required
I0110 23:11:42.054732       5 event.go:218] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"7539f546-f599-11e7-bee6-fa163e2f1153", APIVersion:"v1", ResourceVersion:"127044", FieldPath:""}): type: 'Normal' reason: 'UPDATE' ConfigMap ingress-nginx/nginx-configuration
I0110 23:11:42.138901       5 controller.go:220] ingress backend successfully reloaded...
127.0.0.1 - [127.0.0.1] - - [10/Jan/2018:23:11:56 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "curl/7.47.0" 86 0.003 [upstream-default-backend] 10.244.3.101:80 16 0.003 200
142.xx.xxx.xx - [142.xx.xxx.xx] - - [10/Jan/2018:23:15:50 +0000] "GET / HTTP/1.1" 500 21 "-" "curl/7.47.0" 78 0.020 [upstream-default-backend] 10.244.3.101:80 21 0.020 500
142.xx.xxx.xx - [142.xx.xxx.xx] - - [10/Jan/2018:23:16:02 +0000] "GET /platform/bitcoin HTTP/1.1" 200 45 "-" "curl/7.47.0" 94 0.165 [upstream-default-backend] 10.244.3.101:80 45 0.165 200
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:16:16 +0000] "GET / HTTP/1.1" 500 21 "-" "curl/7.54.0" 78 0.002 [upstream-default-backend] 10.244.3.101:80 21 0.002 500
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:16:30 +0000] "GET /platform/bitcoin HTTP/1.1" 200 45 "-" "curl/7.54.0" 94 0.002 [upstream-default-backend] 10.244.3.101:80 45 0.002 200
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:16:43 +0000] "GET /platform/bitcoin HTTP/1.1" 200 45 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 370 0.049 [upstream-default-backend] 10.244.3.101:80 45 0.049 200
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:16:44 +0000] "GET /favicon.ico HTTP/1.1" 404 9 "http://142.xx.xxx.xx/platform/bitcoin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 324 0.013 [upstream-default-backend] 10.244.3.101:80 9 0.013 404
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:17:04 +0000] "GET /platform/bitcoin HTTP/1.1" 200 45 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 370 0.002 [upstream-default-backend] 10.244.3.101:80 45 0.002 200
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:17:07 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 367 0.002 [upstream-default-backend] 10.244.3.101:80 16 0.002 200
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:17:56 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "curl/7.54.0" 91 0.002 [upstream-default-backend] 10.244.3.101:80 16 0.002 200
Logs from 1/10/18 10:17 PM to 1/10/18 11:17 PM UTC

142.xx.xxx.xx是HAProxy虚拟机的IP

216.249.49.20是来自大学的外部IP.如您所见，入口Pod可以使用use-proxy-protocol: "true"读取从HAProxy传递的外部IP.

216.249.49.20 is an external IP coming from the university. As you can see, the ingress pod can read external IP's passed from HAProxy with use-proxy-protocol: "true" Just fine.

但是当我卷曲HAProxy vm的地址时，我得到:

But when I curl the address of HAProxy vm, I get:

demonfuse@Williams-MacBook-Pro ~/N/K/NGINX> curl 142.xx.xxx.xx/platform/ping
pong2 10.244.2.6

10.244.2.6是入口Pod的IP. 我有信心ingress-nginx拥有真正的源IP.

10.244.2.6 is the IP of the ingress pod. I am confident ingress-nginx at this point has the real source IP.

是否可以通过配置映射将标头和真实源IP转发到ingress-nginx后面的Pod?据我所知此处，大多数情况下默认情况下应处于打开状态

Is there a way to forward the headers and real source IP to pods behind ingress-nginx via configmaps? From what I can tell here it most of it should be turned on by default.

如何复制:

  1. 按照指南在新群集上安装ingress-nginx在这里
  2. 将流量从HAProxy/外部负载均衡器重定向到ingress-nginx
  3. 执行脚本

如下:

import (
        "github.com/kataras/iris"
        "github.com/kataras/iris/context"
        //...
    )

    func main() {
        app := iris.New()
            app.Get("/platform/ping", func(ctx context.Context) {
            fmt.Println("connected with " + ctx.RemoteAddr() + "!")
            ctx.WriteString("pong2 " + ctx.RemoteAddr())
        })

        //...

        app.Run(iris.Addr(":80"), iris.WithoutServerError(iris.ErrServerClosed))
    }

其他信息:

环境:Internet -> Dedicated HAProxy VM -> Bare metal OVH K8S Cluster (1 master, 2 worker)

configmap.yaml

configmap.yaml

apiVersion: v1
data:
  proxy-set-headers: "ingress-nginx/custom-headers"
  use-proxy-protocol: "true"
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app: ingress-nginx

custom_headers.yaml

custom_headers.yaml

apiVersion: v1
data:
  X-Forwarded-For: "142.xx.xxx.xxx"
kind: ConfigMap
metadata:
  name: custom-headers
  namespace: ingress-nginx

haproxy配置

global
   maxconn 4096
   log 127.0.0.1 local0 notice
   maxconn 2000
   user haproxy
   group haproxy

defaults
   log   global
   mode   http
   retries   3
   option redispatch
   maxconn   2000
   timeout connect 5000
   timeout client  50000
   timeout server  50000

frontend TestServerTest
    bind 142.xx.xxx.xxx:80
    mode tcp
    default_backend TestServernodes

backend TestServernodes
    mode tcp
    server TestServer01 142.xx.xxx.xxx:80 send-proxy

我在哪里以及如何出错?

Where and how did I made a mistake?

我尝试将X-Forwaded-For与内部入口Pod IP，与入口服务相关联的外部IP以及HAProxy vm的公共IP结合使用.到目前为止，卷曲HAProxy的外部IP仍会返回pong2 10.244.2.6(入口Pod的内部IP)

I have attempted a combination of X-Forwaded-For with the internal ingress pod IP, the external IP associated with the ingress service, and the public IP of the HAProxy vm. So far curling the external IP of the HAProxy still returns pong2 10.244.2.6 (internal IP of the ingress pod)

推荐答案

我知道了！问题在于Iris Web框架，与ingress-nginx几乎没有关系.

I figured it out! The problem lies in the Iris web framework and has little to nothing to do with ingress-nginx.

解决方案是在ctx.Application().ConfigurationReadOnly().GetRemoteAddrHeaders()中手动读取远程标头.默认情况下，Iris框架不检查X-Forwarded-For和X-Real-Ip

The solution is to read the remote headers manually in ctx.Application().ConfigurationReadOnly().GetRemoteAddrHeaders(). By default the Iris framework does not check for X-Forwarded-For and X-Real-Ip

希望这对于那些往返于Kubernetes的反向代理有用.

Hopefully this will be useful for those running reverse proxies to and from Kubernetes.

这篇关于Kubernetes Ingress-nginx保留源IP的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
其他开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆