Skip to content

Debug Traefik Ingress Controller

Chains of iptables used by Kubernetes

Several chains of iptables are programed to do all kinds of filtering and NAT between pods and services when a Service or Endpoint object is created.

  • KUBE-SERVICES is the entry point for service packets. What it does is to match the destination IP:port and dispatch the packet to the corresponding KUBE-SVC-* chain.
  • KUBE-SVC-* acts as a load balancer, which distributes the packet to KUBE-SEP-* chain. The number of KUBE-SEP-* is equal to the number of endpoints behind the service. Which KUBE-SEP-* to be chosen is determined randomly.
  • KUBE-SEP-* represents a Service EndPoint. It simply does DNAT, replacing service IP:port with pod’s endpoint IP:Port.
  • See: https://serenafeng.github.io/2020/03/26/kube-proxy-in-iptables-mode/

Traefik Status

  • Traefik loadbalancer sould be already configured  for following ports:
    • 8087 -> keycloak Service
    • 80 -> HTTP access
    • 443 -> HTTPS access
    • 8080 -> Traefik Admin port
  • The loadbalancer should route any traefik packages to the traefik loadbalancer running on ip 10.43.99.34
  • The svclb-traefik_XXX pods  are runnng an iptables command to trigger this routing
  • The svclb-traefik_XXX pods are runing on each cluster node
  • Each svclb-traefik_XXX pod runs  4 containers  [ lb-port-80, lb-port-443, lb-port-8080, lb-port-8087 ]
  • Traefik pod is listening on pod 8087
# kubectl get pod -n  traefik-ns  -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP            NODE           NOMINATED NODE   READINESS GATES
svclb-traefik-8rl5h        4/4     Running   0          23h   10.42.0.163   kube-master    <none>           <none>
svclb-traefik-nds69        4/4     Running   0          23h   10.42.2.246   kube-worker2   <none>           <none>
traefik-59cfdc49c7-dmm58   1/1     Running   0          23h   10.42.2.245   kube-worker2   <none>           <none>
svclb-traefik-mk54q        4/4     Running   0          23h   10.42.1.31    kube-worker1   <none>           <none>

# kubectl get svc  -n  traefik-ns  -o wide
NAME      TYPE           CLUSTER-IP    EXTERNAL-IP                                    PORT(S)                                                    AGE   SELECTOR
traefik   LoadBalancer   10.43.99.34   141.75.148.138,141.75.148.139,141.75.148.140   80:31898/TCP,8080:32676/TCP,8087:30158/TCP,443:31879/TCP   27h   app=traefik

# kubectl get endpoints -n  traefik-ns  -o wide
NAME      ENDPOINTS                                                     AGE
traefik   10.42.2.245:80,10.42.2.245:8087,10.42.2.245:443 + 1 more...   27h

Validate Traefik Pod status
# kubectl exec --stdin --tty traefik-59cfdc49c7-dmm58 -n traefik-ns sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

/ # netstat -tulpen  |  egrep '8087 |State'
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 :::8087                 :::*                    LISTEN      1/traefik

Package routing via iptables to traefik IP:Port

  • URL: kube-master.informatik.fh-nuernberg.de:8087
  • iptables routing to keycloak pod running kubeworker2 wiht IP:Port: 10.42.2.245:8087
root@kube-master:/home/master#  iptables -t nat -L KUBE-SERVICES -n  | column -t |  egrep 'KUBE-SERVICES|traefik'
Chain                      KUBE-SERVICES  (2   references)
KUBE-MARK-MASQ             tcp            --   !10.42.0.0/16  10.43.99.34     /*  traefik-ns/traefik:admin                                                      cluster       IP          */     tcp   dpt:8080
KUBE-SVC-2NGTPF5PU2OO4NFC  tcp            --   0.0.0.0/0      10.43.99.34     /*  traefik-ns/traefik:admin                                                      cluster       IP          */     tcp   dpt:8080
KUBE-FW-2NGTPF5PU2OO4NFC   tcp            --   0.0.0.0/0      141.75.148.138  /*  traefik-ns/traefik:admin                                                      loadbalancer  IP          */     tcp   dpt:8080
KUBE-FW-2NGTPF5PU2OO4NFC   tcp            --   0.0.0.0/0      141.75.148.139  /*  traefik-ns/traefik:admin                                                      loadbalancer  IP          */     tcp   dpt:8080
KUBE-FW-2NGTPF5PU2OO4NFC   tcp            --   0.0.0.0/0      141.75.148.140  /*  traefik-ns/traefik:admin                                                      loadbalancer  IP          */     tcp   dpt:8080
KUBE-MARK-MASQ             tcp            --   !10.42.0.0/16  10.43.99.34     /*  traefik-ns/traefik:websecure                                                  cluster       IP          */     tcp   dpt:443
KUBE-SVC-CATTP5HEIU3VUDDR  tcp            --   0.0.0.0/0      10.43.99.34     /*  traefik-ns/traefik:websecure                                                  cluster       IP          */     tcp   dpt:443
KUBE-FW-CATTP5HEIU3VUDDR   tcp            --   0.0.0.0/0      141.75.148.138  /*  traefik-ns/traefik:websecure                                                  loadbalancer  IP          */     tcp   dpt:443
KUBE-FW-CATTP5HEIU3VUDDR   tcp            --   0.0.0.0/0      141.75.148.139  /*  traefik-ns/traefik:websecure                                                  loadbalancer  IP          */     tcp   dpt:443
KUBE-FW-CATTP5HEIU3VUDDR   tcp            --   0.0.0.0/0      141.75.148.140  /*  traefik-ns/traefik:websecure                                                  loadbalancer  IP          */     tcp   dpt:443
KUBE-MARK-MASQ             tcp            --   !10.42.0.0/16  10.43.99.34     /*  traefik-ns/traefik:web                                                        cluster       IP          */     tcp   dpt:80
KUBE-SVC-6R2DDDBNAG6WBID2  tcp            --   0.0.0.0/0      10.43.99.34     /*  traefik-ns/traefik:web                                                        cluster       IP          */     tcp   dpt:80
KUBE-FW-6R2DDDBNAG6WBID2   tcp            --   0.0.0.0/0      141.75.148.138  /*  traefik-ns/traefik:web                                                        loadbalancer  IP          */     tcp   dpt:80
KUBE-FW-6R2DDDBNAG6WBID2   tcp            --   0.0.0.0/0      141.75.148.139  /*  traefik-ns/traefik:web                                                        loadbalancer  IP          */     tcp   dpt:80
KUBE-FW-6R2DDDBNAG6WBID2   tcp            --   0.0.0.0/0      141.75.148.140  /*  traefik-ns/traefik:web                                                        loadbalancer  IP          */     tcp   dpt:80
KUBE-MARK-MASQ             tcp            --   !10.42.0.0/16  10.43.99.34     /*  traefik-ns/traefik:keycloak                                                   cluster       IP          */     tcp   dpt:8087
KUBE-SVC-O2EOQBSIC5NZRIQN  tcp            --   0.0.0.0/0      10.43.99.34     /*  traefik-ns/traefik:keycloak                                                   cluster       IP          */     tcp   dpt:8087
KUBE-FW-O2EOQBSIC5NZRIQN   tcp            --   0.0.0.0/0      141.75.148.138  /*  traefik-ns/traefik:keycloak                                                   loadbalancer  IP          */     tcp   dpt:8087
KUBE-FW-O2EOQBSIC5NZRIQN   tcp            --   0.0.0.0/0      141.75.148.139  /*  traefik-ns/traefik:keycloak                                                   loadbalancer  IP          */     tcp   dpt:8087
KUBE-FW-O2EOQBSIC5NZRIQN   tcp            --   0.0.0.0/0      141.75.148.140  /*  traefik-ns/traefik:keycloak                                                   loadbalancer  IP          */     tcp   dpt:8087

root@kube-master:/home/master# iptables  -v -t nat -L  KUBE-SVC-O2EOQBSIC5NZRIQN  -n  | column -t
Chain  KUBE-SVC-O2EOQBSIC5NZRIQN  (5                         references)
pkts   bytes                      target                     prot         opt  in  out  source     destination
1      60                         KUBE-SEP-AFDBSPUPSG7DXKN6  all          --   *   *    0.0.0.0/0  0.0.0.0/0    /*  traefik-ns/traefik:keycloak  */

root@kube-master:/home/master#  iptables  -v -t nat -L  KUBE-SEP-AFDBSPUPSG7DXKN6   -n  | column -t
Chain  KUBE-SEP-AFDBSPUPSG7DXKN6  (1              references)
pkts   bytes                      target          prot         opt  in  out  source       destination
0      0                          KUBE-MARK-MASQ  all          --   *   *    10.42.2.245  0.0.0.0/0    /*  traefik-ns/traefik:keycloak  */
1      60                         DNAT            tcp          --   *   *    0.0.0.0/0    0.0.0.0/0    /*  traefik-ns/traefik:keycloak  */  tcp  to:10.42.2.245:8087

Links

Published inTraefik

Be First to Comment

Leave a Reply

Your email address will not be published. Required fields are marked *