Alerting: condition precedence order is not expected

[evan361425]

Why is this needed:

When computing multiple alert rules in Grafana, the current logic evaluates all rules before determining whether to fire an alert based on their "and/or" relationships. For example, given four rules with the relationship:

• rule1: result true, operator N/A(first rule don't need an operator)
• rule2: result true, operator and
• rule3: result false, operator or
• rule4: result false, operator and

the algorithm computes as follow (excerpt from pkg/expr/classic/classic.go):

func (cmd *ConditionsCmd) exec() {
	for i, cond := range cmd.Conditions {
		isCondFiring, isCondNoData, condMatches, err := cmd.executeCond(ctx, t, cond, vars)
		if err != nil { return }

		if i == 0 { // first rule have no operator
			isFiring = isCondFiring
			isNoData = isCondNoData
		} else {
			isFiring = compareWithOperator(isFiring, isCondFiring, cond.Operator)
			isNoData = compareWithOperator(isNoData, isCondNoData, cond.Operator)
		}

		matches = append(matches, condMatches...)
	}
}

func compareWithOperator(b1, b2 bool, operator ConditionOperatorType) bool {
	if operator == "or" {
		return b1 || b2
	} else {
		return b1 && b2
	}
}

This computation yields a result of false because:

• true AND true equals to true
• true OR false equals to true
• true AND false equals to false

I expected the computation is same as the logical order: true && true || false && false, resulting in true.

func True() bool {
	print("True, ")
	return true
}
func False() bool {
	print("False, ")
	return false
}

func main()  {
	print(True() && True() || False() && False()) // True, True, true
}

What would you like to be added:

To enhance the logic flow, we can implement a check to return true if we encounter true before an or operation:

for i, cond := range cmd.Conditions {
	if cond.Operator == "or" && isFiring {
		break
	}
	// ...
}

Introducing a new operator, perhaps named lowerPrecedenceOr, could facilitate this enhancement without causing breaking changes.

Who is this feature for?

This feature is essential for users implementing Google's The Site Reliability Workbook Multiple Burn Rate Alerts.
In production, missing alerts due to the following scenario has highlighted the necessity for improvement:

• Alert structure: {short_term} OR {mid_term} OR {long_term}
• Example scenario:
  • The {short_term} rule was triggered as {5_minutes} AND {1_hour}, resulting in true AND true.
  • However, the {mid_term} rule failed to trigger because of {30_minutes} AND {6_hours}, which evaluates to true AND false.
  • Similarly, the {long_term} rule also failed to trigger because of {6_hours} AND {72_hours}, resulting in false AND false.
  • Getting the false result and the alert didn't fired.
• After the improvement, the alert should be fired.

Improving this functionality is crucial for ensuring timely and accurate alerting in complex operational environments.