Topology Aware Routing not working #124699
Labels
kind/bug
Categorizes issue or PR as related to a bug.
needs-triage
Indicates an issue or PR lacks a `triage/foo` label and requires one.
priority/awaiting-more-evidence
Lowest priority. Possibly useful, but not yet enough support to actually get it done.
sig/network
Categorizes an issue or PR as relevant to SIG Network.
What happened?
Despite enabling topology-aware routing, I'm still seeing traffic being routed from one AZ to another. This is not the expected behavior, as it should prioritize in-zone traffic.
Current setup:
3 AZ eu-west
3 pods in each AZ
3 endpoints per each AZ
What did you expect to happen?
When i enable topology aware routing I expect that traffic will stay in AZ where traffic was generated to maximize cost savings.
How can we reproduce it (as minimally and precisely as possible)?
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: nginx
apiVersion: v1
kind: Service
metadata:
annotations:
service.kubernetes.io/topology-aware-hints: Auto
name: nginx
spec:
ports:
protocol: TCP
targetPort: 80
selector:
app: nginx
type: ClusterIP
Anything else we need to know?
kubectl get events --field-selector involvedObject.kind=Service,involvedObject.name=nginx
16m Normal TopologyAwareHintsEnabled service/nginx Topology Aware Hints has been enabled, addressType: IPv4
Kube-Proxy logs:
I0505 16:46:38.759957 10 topology.go:171] "Skipping topology aware endpoint filtering since one or more endpoints is missing a zone hint"
I0505 16:46:38.105331 10 topology.go:171] "Skipping topology aware endpoint filtering since one or more endpoints is missing a zone hint"
I0505 16:49:27.896972 10 topology.go:181] "Skipping topology aware endpoint filtering since no hints were provided for zone" zone="eu-west-1c"
I0505 16:49:28.432488 10 topology.go:181] "Skipping topology aware endpoint filtering since no hints were provided for zone" zone="eu-west-1c"
Hints were assigned:
addressType: IPv4
apiVersion: discovery.k8s.io/v1
endpoints:
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-06c703e12f8392a11
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-jdnd9
namespace: default
uid: a142d4d9-e078-44e9-9ce1-3956faa1e5ce
zone: eu-west-1c
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-06c703e12f8392a11
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-cr7f2
namespace: default
uid: a2bd185c-86d0-47a8-a5ab-45ff4e32a85d
zone: eu-west-1c
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-088e764bd881db13b
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-znbbm
namespace: default
uid: 204eaf01-79bb-4993-a37c-b7bb90fb2094
zone: eu-west-1b
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-088e764bd881db13b
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-lfxqp
namespace: default
uid: 47909cec-d536-4bee-ac95-5b021a29041b
zone: eu-west-1b
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-05ab65b1b0ba1ab03
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-mjvd5
namespace: default
uid: 777dceea-a87f-483d-ac7f-d30cd7812bb1
zone: eu-west-1a
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-05ab65b1b0ba1ab03
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-ltj2v
namespace: default
uid: 2a4d680c-8822-4a2c-b3b8-a56ceaff96e7
zone: eu-west-1a
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-06c703e12f8392a11
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-xwqlm
namespace: default
uid: b4fb49ed-630b-4f7c-8ea0-22cc11ab2c9d
zone: eu-west-1c
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-088e764bd881db13b
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-p92c5
namespace: default
uid: 48c29d22-98f4-4316-bc96-958d608a8ffc
zone: eu-west-1b
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-0272547448c4bb6b7
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-sqb79
namespace: default
uid: 3f206f30-184c-4c83-986f-29b7b4ad5f77
zone: eu-west-1a
kind: EndpointSlice
Kubernetes version
1.26.12
Cloud provider
AWS
OS version
Install tools
Container runtime (CRI) and version (if applicable)
Related plugins (CNI, CSI, ...) and versions (if applicable)
The text was updated successfully, but these errors were encountered: