Hi Dojo,
I noticed something strange with the Correlation Engine for which I don't know if this is default behavior.
My setup has a Correlation Rule that executes a script when an alarm occurs on a spine switch element.
When I create 3 alarms on my element, by changing the Interfaces Admin Status, I notice that the script is only executed once.
Only when I enable option "Trigger on single events. Don't maintain active tree status." my script is executed for every change. But in that case it's also executed every time the alarm is cleared (I know that I can work around this by filtering on Alarm Severity).
My question is how far it's normal behavior that my script is only executed once while 3 unique alarms are created?
All of this is by design.
For a correlation rule, all active alarm trees which match the specified alarm filter are being kept track of in a bucket. In your example, all alarm trees elements running the Arista Switch protocol will be kept track of in one single bucket.
For a bucket, the correlation engine rules execute their actions when the rule condition starts being true for the set of alarm trees inside. (if there are no conditions specified, the condition is true as soon as there is one alarm tree in the bucket). Clear actions (if any) are executed once the rule condition stops being true. If extra alarms are being added to (or removed from) the bucket while the condition is fulfilled, only actions marked as "Execute on base alarm updates" will execute.
In your case, the script will run when the set of active Arista Switch alarm trees contains at least one alarm on the "*SPINE*" element.
- On your first alarm, the rule will fire.
- On the second alarm, the rule will not fire as the bucket already fulfilled the condition
- On the third alarm, the rule will not fire as the bucket already fulfilled the condition
Note that in your case you're only filtering on the protocol. This might cause other unwanted alarms to also be tracked in the bucket. Moving the "*SPINE*" element filter to the alarm filter section could help there.
Note that you can split up the incoming alarms into multiple buckets through the "group by" option, e.g. by element or by table index.
The "trigger on single events" option can also be used if the state keeping is not required (combined with a more precise alarm filter to prevent the same alarm tree from triggering the actions multiple times)
Hi Wouter, thank you for the very detailed explanation. Much appreciated!