Otel Metrics Not Received

[stephen-galea-weavr]

What happened?

I am using Grafana Agent as an otel collector and is working for other otel sources. However when I set a MeshMetric based on (https://kuma.io/docs/2.7.x/guides/otel-metrics/) I am not receiving any metrics.

This is my config:

apiVersion: kuma.io/v1alpha1
kind: MeshMetric
metadata:
  name: otel-metrics
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Mesh
  default:
    # sidecar:
    #   profiles:
    #     appendProfiles:
    #     - name: None
    #     include:
    #     - type: Regex
    #       match: envoy_cluster_external_upstream_rq_.*
    backends:
      - type: OpenTelemetry
        openTelemetry:
          endpoint: grafana-agent-alloy.grafana.svc:4317

I can also confirm that the sidecar is being configured to send metrics

2024-04-24T15:52:48.293Z	INFO	mesh-metric-config-fetcher	Starting OpenTelemetry exporter	{"backend": "grafana-agent-alloy.grafana.svc-4317"}

[Automaat]

@stephen-galea-weavr could you share more info about your Grafana agent setup?

I've just tested it, with the basic setup from Grafana docs, and it works for me

[stephen-galea-weavr]

Hi, sure this is my setup. This is working fine for all other microservices etc... in the cluster.

otelcol.receiver.otlp "default" {
        grpc {
          endpoint = "0.0.0.0:4317"
        }
        http {
          endpoint = "0.0.0.0:4318"
        }
        output {
          traces  = [otelcol.processor.batch.default.input,otelcol.connector.servicegraph.default.input]
          metrics = [otelcol.processor.batch.default.input]
        }
      }

      otelcol.processor.batch "default" {
        output {
          traces  = [otelcol.exporter.otlp.default.input]
          metrics = [otelcol.exporter.prometheus.default.input]
        } 
      }

      otelcol.connector.servicegraph "default" {
        dimensions = ["http.method"]
        output {
            metrics = [otelcol.exporter.prometheus.default.input]
        }
      }

      otelcol.exporter.prometheus "default" {
        forward_to = [prometheus.remote_write.mimir.receiver]
        add_metric_suffixes = false
      }