gke static ip

[oakmad]

We're migrating to gke and have an issue with our ingress. Followed the example gce deployment and it was amazingly easy, everything came up fine, certificate issued no trouble - kudos. We just got the ephemeral ip with that setup and of course it changed after a few days of testing deployments etc, breaking the site.

We grabbed a static ip with

gcloud compute addresses create ip-address --global

and then in my ingress I added

kubernetes.io/ingress.global-static-ip-name: "ip-address"

and applied the ingress again. Here is the complete ingress-tls.yaml

kind: Ingress
metadata:
  name: test-nodejs
  namespace: test-nodejs
  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "gce"
    kubernetes.io/ingress.global-static-ip-name: "ip-address"
spec:
  tls:
  - hosts:
    - test.site.com
    secretName: test-nodejs-tls
  rules:
  - host: test.site.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: test-nodejs
          servicePort: 80

Unfortunately this didn't work with errors in my console "Some backend services are in UNHEALTHY state" on the ingress and the site now returning 502 error. In searching I found this issue kubernetes/ingress-nginx#1281 which states only a regional ip can be used, which I've tried with same result.

Just wondering if I missed something in setting this up? Does anyone have a working static ingress they might share? Might be worth adding this to the example? I'll happily pull request when I sort it out.

Thanks!

[EamonKeane]

I spent some quality time with this issue, as I'm new to Kubernetes. Ultimately I ended up with the below which may be acceptable:

• Installing an nginx controller with 'helm install stable/nginx-ingress' (nginx has additional configuration options such as allowing the load balancer to access ClusterIPs)
• Find the ephemeral IP address created by the controller: Note the nginx-controller pod created (helm will print it out). Then use 'kubectl --namespace default get services -o wide -w your_value-your_value-nginx-ingress-controller'
• Navigating to GCP dashboard > VPC-network > external IP addresses and changing the IP address from ephemeral to static
• In your annotations, use 'kubernetes.io/ingress.class: nginx' instead of gce

This tutorial is also helpful. https://github.com/lachie83/croc-hunter/blob/master/DEMO.md

[oakmad]

Thanks @EamonKeane. I tried bringing up a clean container in case of some legacy issues interfering and got the same result. Now I'm going to try bringing ip up with an ephemeral ip, and switching that to static rather than using my preexisting static address... Failing that then I'll look your solution in more depth. I'm new to kubernetes also, and would like to understand this better. To be honest the ability to easily set a static ip seems like something everyone would require at some point, I'd like to see it clearly documented.

UPDATE: Gah so changing an ip from ephemeral to static also seems to break the ingress. I really don't understand why.

[EamonKeane]

Unfortunate... here's a video (as recent as July) of Google admitting Ingress are not clear.
https://www.youtube.com/watch?v=NafjRSqpFxk

I didn't find any good GKE Ingress IP demos or explanations out there.

[oakmad]

Thanks again @EamonKeane. Yes its definitely murky. The current state is we've totally broken it, even trying to bring it up with a ephemeral address is returning a nginx 502 error. Though I suspect this has something to do with GCP returning the same address that I previously flagged as static. Its all turned rather pear shaped. We're investigating options for GCP or someone just to set it all up for us - we've wasted far too much time on what really should be a trivial exercise. Thanks for your help!

[EamonKeane]

Hi oakmad, I feel your pain. I did some further digging and found a repo by someone at Google from 2 months ago. I ran it on 1.7.8 and it actually seems to work. I haven't applied TLS yet, but you could verify this initially that it works.

https://github.com/cgrant/global-k8s-ingress-with-gce-controller

You can verify it for yourself if you quickly just apply ingress.yaml and simple-echo.yaml without going through the other steps.

I'm not sure what I did differently to what he has here... I think I tried all possible combinations.
Run:
gcloud compute addresses create ingress-ip --global
kubectl apply -f deploy/ingress.yaml
kubectl apply -f deploy/simple-echo.yaml

[EamonKeane]

On further exploration, let me walk back my endorsement of Google working.

After thoroughly checking across several clusters and adding in the setting config.LEGO_LOG_LEVEL=debug, I suspect the issue is Google but accept it could be me.

A good way for people to check this process is to install the Jenkins chart. This is one of the few that has pre-configured options to allow you to add TLS in the values.yaml (you can follow along with this demo here https://github.com/lachie83/croc-hunter/blob/master/DEMO.md).

helm --namespace jenkins --name jenkins -f ./jenkins-values.yaml install stable/jenkins

Add your domain in the hostname of the jenkins-values file. This works perfectly if I use ClusterIP with the nginx ingress.class. Changing to NodePort, ingress.class gce and global static IP results in a single unhealthy backend (wait 5 minutes as this is how long Google's health checks take). For the first 5 minutes they will all be unhealthy. You will then see two healthy backends and one, for no apparent reason, unhealthy. The jenkins url does not load.

A benefit of using the Nginx controller over gce is that it creates fewer backend services vs GCE. They charge for these and impose a cap of 5 BE services for new customers where you have to request to get a higher quota.

If Google were able to provide a fully working example with TLS (perhaps with a random long domain name), this would eliminate any concerns. A further way to check this would be to get an SSL cert the old fashioned way and store it as a secret. @oakmad please post back here if you get any satisfaction from Google!

[oakmad]

Hi @EamonKeane, after a bunch of investigation it turns out we made an error on our image deployment which was muddying things even further. Current status is we have everything working again with an ephemeral ip address. Now just waiting for them to instruct on the best way to use a static ip with the lego ingress. Will update when I hear back.

[oakmad]

Just to update this issue. Google GKE support came back with instructions on how to set a static IP up. We've followed them and so far so good.

First we set the site with a ephemeral address, and then changed that (through the console) to static and gave it a short name e.g. ip-address-name.

Update the echoserver\ingress-tls.yaml file to reference that address: add the following line to the metadata section

kubernetes.io/ingress.global-static-ip-name: "ip-address-name"

here is our live version:

kind: Ingress
metadata:
  name: echoserver
  namespace: echoserver
  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "gce"
    kubernetes.io/ingress.global-static-ip-name: "ip-address-name"
spec:
  tls:
  - hosts:
    - echo.example.com
    secretName: echoserver-tls
  rules:
  - host: echo.example.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: echoserver
          servicePort: 80 

Reapply the ingress with:

kubectl apply -f echoserver/ingress-tls.yaml

This has worked for us todate, but I'll update if we have any other issues.

[EamonKeane]

Hi Oakmad, thanks for posting back. Good to hear there is a solution. Does this give you a regional static IP address as opposed to a Global Static? Is global static possible? What is the practical difference between regional static and global static?

If my intuition is correct regarding regional vs global e.g. a kubernetes cluster in us-west-1 couldn't communicate with a Cloud-SQL database in europe-west-1 via Google's private internet if the IP address is only regional static. I suppose this is possible if IP is global static?
https://cloud.google.com/sql/docs/mysql/connect-container-engine

[oakmad]

Thanks for sticking with this @EamonKeane. When I run gcloud compute addresses describe on the address it shows up as global not as a regional one. We are running cloud-sql proxy from one of our containers and its working fine, so I'm assuming here that its OK now. To be sure, I've forwarded on your intuition to my support person for clarification and will let you know what they come back with.

[oakmad]

HI @EamonKeane, here the response from GKE support on the regional vs global ip.

HTTP(s) load balancer are global resources[1], therefore the instance ephemeral IP that you promoted to static is listed as Global IP[2]. Regarding the connection to Cloud SQL instances, since you need to communicate via the external IP of the Cloud SQL instance, the traffic will be routed to the external internet, regardless if you have an HTTP(s) load balancer with a global IP or a network load balancer[3] which uses a regional IP. Since the traffic is external to a Cloud SQL instance, on GKE it's recommended to use the Cloud SQL proxy[4]. At this other link[5] you can find more information about the network egress pricing.
[1] https://cloud.google.com/compute/docs/load-balancing/http/
[2] https://cloud.google.com/compute/docs/ip-addresses/#reservedaddress
[3] https://cloud.google.com/compute/docs/load-balancing/network/
[4] https://cloud.google.com/sql/docs/mysql/sql-proxy
[5] https://cloud.google.com/sql/pricing#pg-storage-networking-prices

[EamonKeane]

Thanks @oakmad . That clarifies most of it. However I have one question regarding how you setup your service. Did you initially have the service set up to be type LoadBalancer? From what I can gather, this is the only way I can create an ephemeral global static IP address. Did you modify that when redeploying? From what I read elsewhere services are required to be NodePort to work with GKE HTTPs loadbalancing.

https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer#step_2_expose_nginx_as_a_service

[oakmad]

Hi @EamonKeane, well we've pulled it up and down a few times as it was a test site. The site that is currently deployed is in that last ingress-tls.yaml I posted. Though this does appear in the google console as a load balancer, under NETWORK SERVICE >> LOAD BALANCING and essentially references the ingress under DISCOVERY & LOAD BALANCING. Hope that makes sense.

[Jokero]

I would not recommend to use gke ingress. I also started with it and faced many problems (for example I was not able to set timeout for WebSocket in yaml configuration which is pretty easy with nginx ingress).
And gke ingress will not work with minikube. So this solution is not portable.

About this:

. Find the ephemeral IP address created by the controller: Note the nginx-controller pod created (helm will print it out). Then use 'kubectl --namespace default get services -o wide -w your_value-your_value-nginx-ingress-controller'
. Navigating to GCP dashboard > VPC-network > external IP addresses and changing the IP address from ephemeral to static
. In your annotations, use 'kubernetes.io/ingress.class: nginx' instead of gce

@EamonKeane Should I then update LoadBalancer service definition (spec.loadBalancerIP) with this static IP? Or it will be always used until I decide to remove the service?

[EamonKeane]

@Jokero This IP address will always remain the same unless you remove and re-install nginx-ingress. However if you do re-install it, then yea updating the spec.loadBalancerIP should do the trick to preserve it and stop you having to change your DNS records.