I have a multi-threaded timer that is processing about 100+ table rows per second. This leads to a lot of API calls every 24 hours which places a lot of stress on the device communicating with DataMiner. After checking the documentation it seems like either decreasing the threadpool size or adding the polling rate (pollingRate:15,3,3) could help to reduce the number of requests being sent to the device over time.
the each parameter also affects how many rows are polled at a time it seems, but based on my calculations of the table rows in the driver vs the time period of 2 minutes, it should be processing 1.1 rows per second, which is not what the device is receiving.
Which method would be more effective? Other than longer overhead are there any other drawbacks that I may not account for when changing these parameters? Any other possible methods that could work better?
Hi Chian Hao,
If the polling is putting too much load on the device there are multiple ways to reduce the load. First would be good to understand what the limitation is of the device, so you can measure if you are not going above this with the default polling configuration of the connector. There are mainly two things that can overload the device. Continuous load and Peak load. If the device is overloaded with continuous load, then you want to review what you are polling and how frequently you are polling the different data. Maybe the device supports SNMP traps or unconsolidated updates which might allow you to reduce the polling and rely on events forwarded by the device. If the device is overloaded by peak load, then you might want to use WireShark to see how the requests are being handled over time and try to reduce the peaks. This can be done by randomizing the start of your timers (docs). You could also work with buffers to split the calls more over time. For example, if you are only doing a retrieval every 24h and it involves multiple calls to retrieve all data I would propose splitting your requests into chunks (e.g. 10 calls at a time) and using a buffer parameter that keeps track of where you left. Then have a fast timer (e.g. every 1 or 5s, depending on the performance of these calls) and retrieve the next chunk of data from the device until everything is collected. To set your data in DM this is something you will need to decide in your use case if it makes more sense to keep everything in memory before storing it into DM parameters/tables or if you can gradually set it to the parameters/tables. Keep in mind you should never save parameters that frequently change, so don't save this buffer parameter if you go for this approach.
I believe it all comes down to the question 'What will overload the device?'. If you see the device is not handling polling well or you need to poll a lot of data to understand the status of the device, it might be interesting to reach out to the vendor to ask them if they can improve the API of their device, so we can get important KPIs updates from the device in a more efficient way.
How many rows are there in table 1000?
The "each" defines that a row should be polled every 2 minutes. If it cannot reach this rate it will start adding the rows to the waiting queue and then poll it as fast as possible to get rid of the waiting queue. However, a row can only be present once in the waiting queue, so I would not expect it to be executed multiple times within the 2 minutes interval. If the thread pool is decreased it's then not getting enough resources and rows are added to the waiting queue, better to decrease the "each" to have less polling.
What are the other parameters 6201->6206 indicating as value?
What do you mean with calls every 24 hours? Is the multi threaded timer being stopped/started in between?
Are timestamps added when a row gets polled? If so, what do you see when it is changing when focussing on one row? Is this changing around every 2 minutes or faster?
Hi, there’s not more that can be done to improve performance. Regards,
Hi Laurens, apologies for the late response.
Table 1000 has about 155 rows in it. I added a pollingRate: 200,1,1 based on the PollingRateCalculator tool and wanted to share some of the results.
The max duration of polling the table is around 2 seconds, with an average duration of 1.1 seconds. The driver also uses a maximum of 63/100 threads available to make API calls.
My main focus was to spread out the API calls over time as there were reports that the device was being overwhelmed with too many calls, as many as 130/second which is about 480,000 in 24 hours. It’s been running for about a week now with no issues, but I’m wondering if there is anything left to improve performance.