Selectable Index Patterns for Elasticsearch DataSources

[hikerspath]

What would you like to be added:
Allow elasticsearch index name selection when creating a query for a panel (similar to namespace selection on a CloudWatch datasource.

Why is this needed:
Presently, when setting up an elasticsearch datasource, you must enter the index name as part of the datasource configuration. This means that even though accessing the same elasticsearch cluster, I need to manually define every single index pattern as its own datasource.

If instead there was only necessity to define the timeseries pattern format (EX (.*)-YYYY.MM.DD), it would be very easy to provide the user with a selectable list of index patterns that match per query using the GET /_aliases endpoint on an elasticsearch cluster, or perhaps one of the /_cat/ endpoints depending on how the information is parsed.

[henrikno]

Came here to report a similar thing. We have 50+ index patterns, and multiple clusters, so having to make one datasource for each means hundreds of datasources (we can script that part, but managing them in the UI is not easy).

Currently, especially with Index Lifecycle Management or aliases you don't even need to specify -YYYY.MM.DD. Elasticsearch is pretty good at skipping querying indices that don't match the time period.

[danielmotaleite]

Same here!
i worked around by creating a logstash-* source, but this also mean that queries are slow, because they will search in all of indexes instead of a very restricted set

[hikerspath]

I would do that, but we have almost 12000 indices and more than 40k shards housing more than 200TB of logs, so that is not realistic.

[hikerspath]

[simianhacker]

After looking the UI and some discussion with our internal users, I think there are two viable options for overriding index pattern:

Option One

For this option, we would add the index pattern to the query options. This would allow the user to override the index pattern for the entire panel, similar to how the Min interval works. This seems like a natural place since there are already query options overriding the datasource options.

Option Two (prefered)

For this options we would add an Index Pattern field next the the Alias pattern. This would allow you to override the index pattern per query and have the added benefit of supporting multiple index patterns on the same visualization. The other benefit of adding this to the query builder is that it will also work in the Explore interface where option one will only be applied to the panel visualizations.

With both of these options I think just having a way to define a wildcard pattern would be sufficient. The index pattern definition with the date pattern isn't really necessary in Elasticsearch 5+ due to optimizations made in that release (and beyond).

My initial thought was to add multiple patterns to the data source with a way to specify the default. This would require a lot more work because we would need to build an index pattern editor within the datasource and then have a way to select different patterns throughout Grafana. Overriding the pattern in the query builder seems like a good compromise.

@Elfo404 Thoughts?

[simianhacker]

For those following along, I started looking at the code today to see what it would take to make this happen. To my surprise it's pretty minimal, you can check out the draft PR here: #29477

[gelicia]

Hello, as you may have heard, we are transitioning away from using discussions to discuss feature requests. We are migrating this discussion to an issue and closing the discussion. The issue is #83012. Feel free to continue the discussion around this there. Thank you!

[cjabrantes]

Hi to all the people in this issue, as this feature got stall, i believe we should make a comment in the new discussion:
#83012

to make some traction on the topic and so to help prioritise this feature.

Thanks,