Hi,
I am looking at improving stability and reliability in our dataminer implementation. Currently we seem to be experiencing some dataloss and deadlocking occasionally during high resource utilization periods when sending data from one element to another element. It's an inconsistent issue so I'm looking broadly across our solution to try see what can be improved to reduce this. Please could I get some info on the following questions:
Question 1: For each element we have a single parameter existing that is responsible for all incoming data from external elements. Would we benefit from creating one I/O parameter for each other element instead of multiple writing to the same parameter?
Question 2: When the external elements write to the parameter, for our implementation they are currently using the "NotifyDataMiner" method, would we benefit from switching to "NotifyDataMinerQueued"? I don't think the docs are clear enough on the differences. Does the underlying dataminer protocol already queue some stuff even when not using the queued version of the function? A bit more context on the pros/cons of using these different methods would be very helpful.
Question 3: When data is being queued on dataminer parameters, is there configuration somewhere that defines the buffer depth and queue behaviour?
Links below for methods referenced in question 2
NotifyDataMiner(int, object, object) - Method NotifyDataMiner | DataMiner Docs
NotifyDataMinerQueued(int, object, object) - Method NotifyDataMinerQueued | DataMiner Docs
Thank you!
Hi Jordan, thank you for this interesting question.
Question 1:
By default, this would not help as, by default, QAction executions are synchronous, only one can run at a time. It could potentially help if each parameter would trigger its own QAction on which you would be adding the queued option. Keep in mind this means several QActions might be running in parallel, meaning you'll need to be sure your code is thread-safe and add some locking if there is a chance that multiple QActions end up interacting with the same protocol parameters. Note that you also need to be very careful that when using the queued option, not only other QActions can run in parallel but also a second instance of the same QAction can also run before the first instance of it is finished and your QAction run will not block subsequent sets on that parameter.
Let's say param 1 triggers QAction 1 which is not queued. If in your code you set that param 1 to A, first QAction instance will start executing. If you right after set that same parameter to value B, such set will wait until your first QAction execution is finished. This mean that when into your QAction 1, you can the protocol.GetParameter method on parameter 1, you are guaranteed to receive value A. Then only when first instance of QAction 1 is finished, the parameter will be set to B and second run of QAction 1 will start.
Let's say param 1 triggers QAction 1 which is queued now. If in your code you set that param 1 to A and then to B, first QAction instance will start executing but your second set to value B will not wait. Meaning that when in first QAction run, you do a GetParameter call on parameter 1, it might already be at value B while you would probably hope to get A. The second run of that QAction will then probably again get value B unless some other sets to it where executed in the meantime.
Question 2:
I agree the docs should be improved there, we'll look into that. The difference is that NotifyDataMiner is synchronous, meaning that when you call it, your code will wait for the set to be finished, you'll typically use this one if you require subsequent code to be executed only when the set fully went through. While the NotifyDataMinerQueued is asynchronous, your call will directly be added to a queue and your code will continue to execute while the set is likely not yet processed, this will typically be preferred if subsequent code does not rely on the fact the set fully went through.
Question 3:
No, the queue will always behave as a queue => FIFO, and there is currently no configuration regarding the buffer depth. There is currently no max depth defined for it.
Extra info:
Note that instead of using those NotifyDataMiner calls, I would advise you to use following NuGet package: NuGet Gallery | Skyline.DataMiner.Core.DataMinerSystem.Protocol 1.1.2.1. The NotifyDataMiner are low level calls which typically will keep the same implementation to keep backward compatibility. The NuGet package mentioned above is high level api containing some safeguards and is maintained to insure best performance. Over time, the same call might be adapted to use newer low level calls for performance or any other reasons.
You'll find more info on how to use it there: Class library introduction | DataMiner Docs
Hope this helps, let us know if you need more.
Here is a quick and dirty sample code on how to set a parameter using the NuGet package:
var dma = protocol.GetDms().GetAgent(1);
var element = dma.GetElement("");
var param = element.GetStandaloneParameter<string>(5);
param.SetValue("");