I have a multi-threaded timer that is processing about 100+ table rows per second. This leads to a lot of API calls every 24 hours which places a lot of stress on the device communicating with DataMiner. After checking the documentation it seems like either decreasing the threadpool size or adding the polling rate (pollingRate:15,3,3) could help to reduce the number of requests being sent to the device over time.
the each parameter also affects how many rows are polled at a time it seems, but based on my calculations of the table rows in the driver vs the time period of 2 minutes, it should be processing 1.1 rows per second, which is not what the device is receiving.
Which method would be more effective? Other than longer overhead are there any other drawbacks that I may not account for when changing these parameters? Any other possible methods that could work better?
Hi Chian Hao,
If the polling is putting too much load on the device there are multiple ways to reduce the load. First would be good to understand what the limitation is of the device, so you can measure if you are not going above this with the default polling configuration of the connector. There are mainly two things that can overload the device. Continuous load and Peak load. If the device is overloaded with continuous load, then you want to review what you are polling and how frequently you are polling the different data. Maybe the device supports SNMP traps or unconsolidated updates which might allow you to reduce the polling and rely on events forwarded by the device. If the device is overloaded by peak load, then you might want to use WireShark to see how the requests are being handled over time and try to reduce the peaks. This can be done by randomizing the start of your timers (docs). You could also work with buffers to split the calls more over time. For example, if you are only doing a retrieval every 24h and it involves multiple calls to retrieve all data I would propose splitting your requests into chunks (e.g. 10 calls at a time) and using a buffer parameter that keeps track of where you left. Then have a fast timer (e.g. every 1 or 5s, depending on the performance of these calls) and retrieve the next chunk of data from the device until everything is collected. To set your data in DM this is something you will need to decide in your use case if it makes more sense to keep everything in memory before storing it into DM parameters/tables or if you can gradually set it to the parameters/tables. Keep in mind you should never save parameters that frequently change, so don't save this buffer parameter if you go for this approach.
I believe it all comes down to the question 'What will overload the device?'. If you see the device is not handling polling well or you need to poll a lot of data to understand the status of the device, it might be interesting to reach out to the vendor to ask them if they can improve the API of their device, so we can get important KPIs updates from the device in a more efficient way.