docs(MADR): add policy matching for MeshService

[michaelbeaumont]

Checklist prior to review

• Link to relevant issue as well as docs and UI issues -- MADR for Policy Matching and Inspect API of MeshService #9707
• This will not break child repos: it doesn't hardcode values (.e.g "kumahq" as a image registry) and it will work on Windows, system specific functions like syscall.Mkfifo have equivalent implementation on the other OS --
• Tests (Unit test, E2E tests, manual test on universal and k8s) --
  • Don't forget ci/ labels to run additional/fewer tests
• Do you need to update UPGRADE.md? --
• Does it need to be backported according to the backporting policy? (this GH action will add "backport" label based on these file globs, if you want to prevent it from adding the "backport" label use no-backport-autolabel label) --

[jakubdyszkiewicz]

how do we select multiple services? Can MeshService just select multiple services or do we introduce MeshServiceGroup?

[michaelbeaumont]

It should be covered by using labels instead of name

[lahabana]

It's also not super clear where you can use a set of services vs a single service

[jakubdyszkiewicz]

Do we need to differentiate it?

[michaelbeaumont]

To be clear, I don't mind if we want to introduce another kind for groups, as long as the naming is consistent.

[lahabana]

I'm concerned with the same targetRef and merging. How do you order policies?

[michaelbeaumont]

If you order MeshServiceGroup before MeshService, you order MeshService + labels before MeshService + name.
There's no information lost between the two representations.

[michaelbeaumont]

This is not as nice as I thought. So we don't want to assume namespace similarity, that's good. But how do you refer to a MeshService from another zone then?

targetRef:
  namespace: other-zone-ns
  labels:
    kuma.io/zone: other-zone

isn't ideal because the real object is not found at this namespace, rather at kuma-system.

so do we have

targetRef:
  namespace: kuma-system
  labels:
    kuma.io/zone: other-zone
    k8s.kuma.io/namespace: other-zone-ns

? This is the cleanest but it's very verbose.

[Automaat]

Shouldn't it be k8s.kuma.io/namespace: other-zone-ns in the second example?

[Automaat]

So this means that you need to create policy in kuma-system namespace in order to target MeshService in different zone? While discussing policies on namespace MADR we talked that in best case scenario SO won't need to apply anything on kuma-system namespace

[lobkovilya]

Well, TBH it's not drastically more verbose than the first examples. I'd stick with the more verbose version in favor of preserving the rule that we reference real objects

[slonka]

I agree with Ilya here ☝️

[michaelbeaumont]

OK spoke with @jakubdyszkiewicz and we decided to just disallow namespace when using labels, and all namespaces will be matched. This makes it more verbose for the local case of a group of MeshServices but 🤷

[michaelbeaumont]

So this means that you need to create policy in kuma-system namespace in order to target MeshService in different zone?

I don't think so, it's just about namespace in the target ref itself. Updated the YAML to make it clearer.

[lahabana]

I think this type of matching makes sense to me. Only thing I'd like discussed is using kind: MeshService as a top level targetRef similar to kind: MeshHTTPRoute. In such case the to[].targetRef.kind would always need to be Mesh. I'd love to see pros and cons of this.
The fact we're moving to MeshService might an opportunity to iron out issues on the policy matching story that exists nowadays.

[lahabana]

I'm starting to like this. I have a bit of a concern now that we're talking about product vs consumer policies that I think we need to give good thoughts on

[lahabana]

Is it worth saying something like: When a targetRef selects multiple MeshServices and ports this behaves as a "foreach" on each individual entity ?

[lahabana]

IIUC when you are trying to match a policy on a namespace in a different zone you'd need to use labels like:

kind: MeshService
labels:
  kuma.io/zone: other-zone
  k8s.kuma.io/namespace: my-namespace
  kuma.io/display-name: the-service-I-consume

Is that correct?

[michaelbeaumont]

yes. our goal was to reserve name/namespace for real objects on this cluster.

[lahabana]

What happens if this matches multiple MeshService? Is it equivalent to having N entries (N is the number of services matched) in backendRefs?

[michaelbeaumont]

And in particular, what happens to weight? Not sure...

[lahabana]

Status on the policy? On the Dataplane?

[michaelbeaumont]

On the policy/route for sure.

[lahabana]

With the producer and consumer model we're building up in: #10148
It's not super clear how things work as now zone policies could be synced to other zones.

1. For example I have a backend service:

kind: MeshService
metadata:
  name: my-backend
  namespace: my-backend-ns
spec:
  selector:
      dataplaneTags:
        app: my-backend
  ports:
    - port: 80
       targetPort: 8080
       name: http
       appProtocol: http

2. Intuitively I'd write this timeout policy that would affect all consumers of my service (in ALL zones):

kind: MeshTimeout
metadata:
  name: my-backend
  namespace: my-backend-ns
spec:
  targetRef:
     kind: Mesh
  to:
    - targetRef:
          kind: MeshService
          name: my-backend
          port: 80 # (subquestion 1: here the alternative sectionName: http is valid too right?)
      default:
         timeout: 5s

3. Then I'd want to write a consumer policy in the same zone as:

kind: MeshTimeout
metadata:
   namespace: my-frontend-ns
   name: override-my-backend
spec:
  targetRef:
     kind: Mesh
  to:
    - targetRef:
          kind: MeshService
          name: my-backend
          namespace: my-backend-ns
          port: 80 # (subquestion 1: here the alternative sectionName: http is valid too right?)
      default:
         timeout: 5s

4. And I'd write a consumer policy in a different zone as:

kind: MeshTimeout
metadata:
   namespace: my-frontend-ns-in-another-zone
   name: override-my-backend
spec:
  targetRef:
     kind: Mesh
  to:
    - targetRef:
          kind: MeshService
          sectionName: http
          labels:
            kuma.io/display-name: my-backend
            k8s.kuma.io/namespace: my-backend-ns
            kuma.io/zone: my-other-k8s-cluster
      default:
         timeout: 10s

1 and 2 gets synced to remote zones. 3 and 4 don't. @lobkovilya is this correct?

Multiple questions come up to me:

• Do we find it acceptable that matching MeshService in the same zone vs cross zone is syntactically very different (Using labels for remote vs using name/ns for local)?
• With producer policies these gets synced to remote zones now. Is it still straight-forward to do the matching (do we use the synced labels to enhance the matcher)?

[lobkovilya]

Policy from the step 2 won't pass the new validation, it is created in my-backend-ns, it targets kind: Mesh, but it doesn't specify k8s.kuma.io/namespace: my-backend-ns and kuma.io/zone: zone-with-backend in each item of to[], so it should be:

kind: MeshTimeout
metadata:
  name: my-backend
  namespace: my-backend-ns
spec:
  targetRef:
     kind: Mesh
  to:
    - targetRef:
          kind: MeshService
          labels:
            kuma.io/display-name: my-backend
            kuma.io/zone: zone-with-backend
            k8s.kuma.io/namespace: my-backend-ns
          port: 80 # (subquestion 1: here the alternative sectionName: http is valid too right?)
      default:
         timeout: 5s

Same for the policy from step 3:

kind: MeshTimeout
metadata:
   namespace: my-frontend-ns
   name: override-my-backend
spec:
  targetRef:
     kind: MeshSubset
     tags:
       k8s.kuma.io/namespace: my-frontend-ns
       kuma.io/zone: zone-with-frontend
  to:
    - targetRef:
          kind: MeshService
          name: my-backend
          namespace: my-backend-ns
          port: 80 # (subquestion 1: here the alternative sectionName: http is valid too right?)
      default:
         timeout: 5s

[lobkovilya]

I'm talking about these rules:

The validation rules for namespace-scoped policy are the following:

• to-policy when placed into the custom namespace must have the namespace and zone either in the top-level targetRef or in each to-item
• from-policy when placed into the custom namespace must always have the namespace and zone in the top-level targetRef