Use point in time values for gauge metrics in graphs

[shtripat]

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers
under the terms of the Apache 2 license.
By creating this pull request I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 license.

Description

Motivation and Context

How to test this PR?

Load the dashboard in grafana

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Optimization (provides speedup with no functional changes)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Fixes a regression (If yes, please add commit-id or PR # here)
• Unit tests added/updated
• Internal documentation updated
• Create a documentation update request here

[poornas]

@shtripat, grafana/grafana#78053 (comment) seems to indicate rate is incorrect to use for gauge as well - can you check?

[shtripat]

@shtripat, grafana/grafana#78053 (comment) seems to indicate rate is incorrect to use for gauge as well - can you check?

Oh, sure. I will check and change accordingly.

[shtripat]

@shtripat, grafana/grafana#78053 (comment) seems to indicate rate is incorrect to use for gauge as well - can you check?

Oh, sure. I will check and change accordingly.

Based on this looks like we should rather render the graphs using metrics as is without ratifying it. The graph can simple show the values over time.
I am not too sure if its hard rule to not use rate as well for gauge metrics. increase makes sense not to use as gauge metrics can go in -ve as well and increase is meant more for calculating the increase rate.

[shtripat]

Some more reading shows findings like

While we can use the rate() function with gauge metrics, it's essential to interpret the results appropriately. Since gauges represent instantaneous values, the rate of change calculated by the rate() function may fluctuate significantly depending on the frequency of scrapes and variations in the underlying metric values. Therefore, it's crucial to consider the context and understand the behavior of the gauge metric when using the rate() function with it in Grafana queries.

based on this, simpler it would be to just graph the metrics values as is which shows point in time values for the metrics.
@poornas suggestions?