Hi Dojo,
We are implementing an OpenTelemetry Collector that sends logs via interapp to a Receiver sometimes to the tune of ~300 messages per min (see statistics screenshot).
We've noticed that after a short time, the messages get stuck in the interapp message "stack", where even if we disable them they will continue coming - likely the interapp buffer catching up.
Couple questions:
-Is there a way to tell how many messages an element can handle at once? We'd like to add some thresholds to control and possibly to alarm when too many messages are coming in.
-Is there a way to STOP sending the interapp messages immediately, regardless of how many are in the stack?
Here is our sender code, for reference:
var interAppMessage = new TelemetryMessage
{
MessageType = elementInfo.MessageType,
Message = HelpMethods.CompressString(parsed.ToString()),
ManagerElement = protocol.DataMinerID + "/" + protocol.ElementID,
FilterKey = elementInfo.FilterKey,
};
IInterAppCall command = InterAppCallFactory.CreateNew();
command.Messages.Add(interAppMessage);
command.Source = new Source("OpenTelemetry Collector", protocol.DataMinerID, protocol.ElementID);
command.ReturnAddress = new ReturnAddress(elementInfo.DmaId, elementInfo.ElementId, Constants.InterappReturnPid);
command.Send(protocol.SLNet.RawConnection, elementInfo.DmaId, elementInfo.ElementId, Constants.InterappReceiverPid, Constants.KnownTypes());
Thanks in advance!
What I expect you are facing is not directly the Interapp "stack" because from what I remember Interapp sets data directly to the element. So what you are actually seeing is probably the element catching up to the sets.
If it's that you can't stop the messages from coming in. I suspect the root cause is actually the processing of the messages, probably this goes to slow.
I faced that before and this is how I handled it:
a message comes in => process it on a different protocol thread
if another message comes in while the processing is being done => put it on a buffer (I keep this buffer and the busy state in memory, FYI I have failsafes if SLScripting crashes best to keep that in mind)
When the processing is done, trigger the buffer QA to send the buffer and process the next messages in bulk.
That way you can very quickly handle the incoming messages.
Working with a system like this will give you also more flexibility and control of the buffer, you will also be able to print more logging to see what is going on and where the delays are
Just a tip: from the moment you handle Interapp or subscriptions, handle them as quickly as possible and do the processing separately.
One small Note: there is also a possibility to send multiple interapp calls in one go/bulk. No idea if that's applicable in your use-case, but it might be one extra thing for you to consider.