How is the localityLbSetting of DR works in envoy config?

[so2bin]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

backgroud

• We have an Istio service mesh with three clusters, and we've configured a DestinationRule (DR) for cross-cluster traffic distribution as follows. Traffic from all three clusters is directed to the A/F cluster in a ratio of A:F=68:31. This configuration applies to the atms-glb-image-auditing-test.ai-ppt-beautify-test service:

spec:
  host: atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local
  trafficPolicy:
    connectionPool:
      http:
        http2MaxRequests: 100
        maxRequestsPerConnection: 100
    loadBalancer:
      localityLbSetting:
        distribute:
        - from: F/*
          to:
            F/*: 32
            A/*: 68
        - from: E/*
          to:
            F/*: 32
            A/*: 68
        - from: A/*
          to:
            F/*: 32
            A/*: 68
    outlierDetection:
      baseEjectionTime: 30s
      consecutive5xxErrors: 1
      interval: 10s

• Currently, there are two pods each for the atms-glb-image-auditing-test service in clusters A and F.

Question

• As I understand it, the weights configured here will ultimately affect the envoy cluster configuration. However, when I checked the results within the Istio Ingress gateway pod, I found the following:

istio-proxy@istio-ingress-78d4dcf484-pxpn9:/$ curl -s http://localhost:15000/clusters | grep "outbound|80|"| grep glb-image-audit | egrep "weight|region|:zone"
outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.40.0.83:15443::weight::2
outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.40.0.83:15443::region::F
outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.40.0.83:15443::zone::zone2
outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.245.198.35:8000::weight::1
outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.245.198.35:8000::region::A
outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.245.198.35:8000::zone::zone1
outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.245.75.47:8000::weight::1
outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.245.75.47:8000::region::A
outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.245.75.47:8000::zone::zone1

• In the output above, we can see three clusters:

1. One cluster routes traffic to the F cluster with a weight of 2, matching the number of pods in cluster F.

outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.40.0.83:15443::weight::2
outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.40.0.83:15443::region::F
outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.40.0.83:15443::zone::zone2

2. The remaining two clusters are local pod IP endpoints in cluster A, each with a weight of 1, matching the number of pods in cluster A.

outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.245.198.35:8000::weight::1
outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.245.198.35:8000::region::A
outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.245.198.35:8000::zone::zone1
outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.245.75.47:8000::weight::1
outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.245.75.47:8000::region::A
outbound|80||atms-glb-image-auditing-test.ai-ppt-beautify-test.svc.cluster.local::10.245.75.47:8000::zone::zone1

• However, I have concerns about the values of weight. The weight distribution here is A:F=2:(1+1), which clearly doesn't match the DR weight of A:F=68:31. Of course, the ultimate effect of this DR configuration, with traffic ratio A:F=68:31, is consistent with the DR, not the 1:1 distribution shown in :15000/clusters.

• So, are the weights configured in localityLbSetting of DR not the same as the weight queried in :15000/clusters? If not, where does localityLbSetting configuration take effect in Envoy, and where can I find it in envoy?

Thanks.

Version

* istio version

client version: 1.22.0
control plane version: 1.21.1
data plane version: 1.21.1 (113 proxies)


* kube version
```bash
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.6", GitCommit:"741c8db18a52787d734cbe4795f0b4ad860906d6", GitTreeState:"clean", BuildDate:"2023-09-13T09:21:34Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.6", GitCommit:"741c8db18a52787d734cbe4795f0b4ad860906d6", GitTreeState:"clean", BuildDate:"2023-09-13T09:14:09Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}

### Additional Information

_No response_

[howardjohn]

#50746 very likely fixes this. Will be in the next 1.22 and 1.21 patch sorry wrong issue

[so2bin]

@ramaraochavali Hello, I see that you implemented the logic for loadbalancer in this code . Could you help me understand this issue?
https://github.com/istio/istio/blob/master/pilot/pkg/networking/core/loadbalancer/loadbalancer.go

[ramaraochavali]

@so2bin Sorry. Little busy with other work. Will respond soon. @hzxuzhonghu if you have some free cycles?

[hzxuzhonghu]

Sure, i would do some investigate