Similar to this question Max Number of Parameter Subscriptions in DMS - DataMiner Dojo
There is a manager element which is configured to receive updates of metrics from multiple elements using subscriptions. These values change every minute or so. About 95% are cell values and 5% stand-alone parameters. At this point the element has about 4500 parameters which is subscribed to. There are some discrepancies between the values the manager element is reporting and the actual element which I'm guessing it has to do with the overall load of the system. As a temporary solution if we spread the load between different elements lets say 4 managers so each of them have about 1000 subscriptions would this improve things? Or since they share the same process would there be no difference? Would it be better to spread the managers on different DMAs within the cluster?
Hi Geovanny,
We had two major improvements on another project in which we did subscriptions from many to many. It is depending on your use case if these will also help your case. Some improvements to consider:
- Subscribing to table updates instead of to the cells themselves. For table subscriptions, you will only receive the rows that have been updated. This way you have fewer subscriptions which means that SLNet needs to go through fewer checks on every table update.
- In our case, we had many elements subscribing to similar parameters. This is why we manage the subscriptions in a static class. From the moment an element is created we initialize a static subscription manager in QActions. The manager handles all subscriptions of all elements running the same protocol & version. So they ask to subscribe to the manager and this one will then evaluate if there is already a subscription to the parameter. The manager will forward the update(s) to all elements that have asked to make a subscription to it.
The last option is quite complex and might take you quite some time to implement. If you go for the last option, we might evaluate if a standard library could be used. This way other projects could reuse the classes.
The manager element gets subscribed to parameters from elements like the Microsoft Platform, CISCO DMC along a few others so adding some logic to the source elements is probably not the way to go. I would keep in mind your other suggestions as I investigate further. Thank you!
Could you provide more context on what you mean with "discrepancies between the values the manager element is reporting"? It's important you can identify the bottleneck, before you start changing anything. The bottleneck could be any (or all) of the following:
- Subscription updates not coming in (or not fast enough)
- Subscription updates not handled fast enough
- Parameters in your manager element not displaying the correct value (fast enough)
Depending on the bottleneck, a different solution might be needed (one of them could be what Michiel provided already).
It actually turns out that the discrepancies that the customer was reporting was not on the values but instead in the number of metrics. To elaborate a little the manager is subscribed to 4000+ metrics/parameters, every 30 seconds an external application sends a request to the element which has a Webhook which responds with all the values from the subscribe parameters in a specific format which then gets pass to Grafana to further process the information. The issue is the element would respond with only 3500 metrics for example when we can see that there are 4000. The issue might not be in the subscriptions them self but on the load that they are putting on the element. Is there a maximum number of subscription that an element should have? I know this could depend on so many factors but is there a recommended value?
In case the cells are really coming from different tables, there might not be that much improvement in subscribing to the complete table. Something else you might then consider is to add logic in the source elements, but if this means updating many protocols, then you might want to raise this topic with our data acquisition domain to evaluate the best option before making changes.