Skip to content
DataMiner DoJo

More results...

Generic selectors
Exact matches only
Search in title
Search in content
Post Type Selectors
Search in posts
Search in pages
Search in posts
Search in pages
Log in
Menu
  • Blog
  • Questions
  • Learning
    • E-learning Courses
    • Open Classroom Training
    • Certification
      • DataMiner Fundamentals
      • DataMiner Configurator
      • DataMiner Automation
      • Scripts & Connectors Developer: HTTP Basics
      • Scripts & Connectors Developer: SNMP Basics
      • Visual Overview – Level 1
      • Verify a certificate
    • Tutorials
    • Video Library
    • Books We Like
    • >> Go to DataMiner Docs
  • Expert Center
    • Solutions & Use Cases
      • Solutions
      • Use Case Library
    • Markets & Industries
      • Media production
      • Government & defense
      • Content distribution
      • Service providers
      • Partners
      • OSS/BSS
    • DataMiner Insights
      • Security
      • Integration Studio
      • System Architecture
      • DataMiner Releases & Updates
      • DataMiner Apps
    • Agile
      • Agile Webspace
      • Everything Agile
        • The Agile Manifesto
        • Best Practices
        • Retro Recipes
      • Methodologies
        • The Scrum Framework
        • Kanban
        • Extreme Programming
      • Roles
        • The Product Owner
        • The Agile Coach
        • The Quality & UX Coach (QX)
    • DataMiner DevOps Professional Program
  • Downloads
  • More
    • Feature Suggestions
    • Climb the leaderboard!
    • Swag Shop
    • Contact
      • General Inquiries
      • DataMiner DevOps Support
      • Commercial Requests
    • Global Feedback Survey
  • PARTNERS
    • All Partners
    • Technology Partners
    • Strategic Partner Program
    • Deal Registration
  • >> Go to dataminer.services

Reduce number of HTTP table rows processed simultaneously in multi-threaded timer

Solved1.11K views18th October 2023HTTP Multi-Threaded Timer Multithreaded Timer threadPool
3
Chian Hao Aw [SLC] [DevOps Advocate]650 18th July 2023 0 Comments

I have a multi-threaded timer that is processing about 100+ table rows per second. This leads to a lot of API calls every 24 hours which places a lot of stress on the device communicating with DataMiner. After checking the documentation it seems like either decreasing the threadpool size or adding the polling rate (pollingRate:15,3,3) could help to reduce the number of requests being sent to the device over time.

the each parameter also affects how many rows are polled at a time it seems, but based on my calculations of the table rows in the driver vs the time period of 2 minutes, it should be processing 1.1 rows per second, which is not what the device is receiving.

Which method would be more effective? Other than longer overhead are there any other drawbacks that I may not account for when changing these parameters? Any other possible methods that could work better?

Marieke Goethals [SLC] [DevOps Catalyst] Selected answer as best 18th October 2023

2 Answers

  • Active
  • Voted
  • Newest
  • Oldest
3
Michiel Saelen [SLC] [DevOps Enabler]5.63K Posted 18th July 2023 0 Comments

Hi Chian Hao,

If the polling is putting too much load on the device there are multiple ways to reduce the load. First would be good to understand what the limitation is of the device, so you can measure if you are not going above this with the default polling configuration of the connector. There are mainly two things that can overload the device. Continuous load and Peak load. If the device is overloaded with continuous load, then you want to review what you are polling and how frequently you are polling the different data. Maybe the device supports SNMP traps or unconsolidated updates which might allow you to reduce the polling and rely on events forwarded by the device. If the device is overloaded by peak load, then you might want to use WireShark to see how the requests are being handled over time and try to reduce the peaks. This can be done by randomizing the start of your timers (docs). You could also work with buffers to split the calls more over time. For example, if you are only doing a retrieval every 24h and it involves multiple calls to retrieve all data I would propose splitting your requests into chunks (e.g. 10 calls at a time) and using a buffer parameter that keeps track of where you left. Then have a fast timer (e.g. every 1 or 5s, depending on the performance of these calls) and retrieve the next chunk of data from the device until everything is collected. To set your data in DM this is something you will need to decide in your use case if it makes more sense to keep everything in memory before storing it into DM parameters/tables or if you can gradually set it to the parameters/tables. Keep in mind you should never save parameters that frequently change, so don't save this buffer parameter if you go for this approach.

I believe it all comes down to the question 'What will overload the device?'. If you see the device is not handling polling well or you need to poll a lot of data to understand the status of the device, it might be interesting to reach out to the vendor to ask them if they can improve the API of their device, so we can get important KPIs updates from the device in a more efficient way.

Marieke Goethals [SLC] [DevOps Catalyst] Selected answer as best 18th October 2023
You are viewing 1 out of 2 answers, click here to view all answers.
Please login to be able to comment or post an answer.

My DevOps rank

DevOps Members get more insights on their profile page.

My user earnings

0 Dojo credits

Spend your credits in our swag shop.

0 Reputation points

Boost your reputation, climb the leaderboard.

Promo banner DataMiner DevOps Professiona Program
DataMiner Integration Studio (DIS)
Empower Katas

Recent questions

Alarm Dashboard PDF/CSV Export 0 Answers | 0 Votes
Is the Microsoft SharePoint Connector Still Usable 0 Answers | 0 Votes
Is the Microsoft SharePoint Connector Still Usable 0 Answers | 0 Votes

Question Tags

adl2099 (115) alarm (62) Alarm Console (82) alarms (100) alarm template (83) Automation (223) automation scipt (111) Automation script (167) backup (71) Cassandra (180) Connector (108) Correlation (68) Cube (150) Dashboard (194) Dashboards (188) database (83) DataMiner Cube (57) DIS (81) DMS (71) DOM (139) driver (65) DVE (55) Elastic (83) Elasticsearch (115) elements (80) Failover (104) GQI (159) HTTP (76) IDP (74) LCA (151) low code app (166) low code apps (93) lowcodeapps (75) MySQL (53) protocol (203) QAction (83) security (88) services (51) SNMP (86) SRM (337) table (54) trending (87) upgrade (62) Visio (539) Visual Overview (345)
Privacy Policy • Terms & Conditions • Contact

© 2025 Skyline Communications. All rights reserved.

DOJO Q&A widget

Can't find what you need?

? Explore the Q&A DataMiner Docs

[ Placeholder content for popup link ] WordPress Download Manager - Best Download Management Plugin