Skip to content
DataMiner DoJo

More results...

Generic selectors
Exact matches only
Search in title
Search in content
Post Type Selectors
Search in posts
Search in pages
Search in posts
Search in pages
Log in
Menu
  • Updates & Insights
  • Questions
  • Learning
    • E-learning Courses
    • Empower Replay: Limited Edition
    • Tutorials
    • Open Classroom Training
    • Agility
      • Kanban workshop
      • Agile Fundamentals
    • Certification
      • DataMiner Fundamentals
      • DataMiner Configurator
      • DataMiner Automation
      • Scripts & Connectors Developer: HTTP Basics
      • Scripts & Connectors Developer: SNMP Basics
      • Visual Overview – Level 1
      • Verify a certificate
    • Video Library
    • Books We Like
    • >> Go to DataMiner Docs
  • Expert Center
    • Solutions & Use Cases
      • Solutions
      • Use Case Library
    • Markets & Industries
      • Media production
      • Government & defense
      • Content distribution
      • Service providers
      • Partners
      • OSS/BSS
    • Agile
      • Agile Webspace
      • Everything Agile
        • The Agile Manifesto
        • Best Practices
        • Retro Recipes
      • Methodologies
        • The Scrum Framework
        • Kanban
        • Extreme Programming
      • Roles
        • The Product Owner
        • The Agile Coach
        • The Quality & UX Coach (QX)
    • DataMiner DevOps Professional Program
      • About the DevOps Program
      • DataMiner DevOps Support
  • Downloads
  • More
    • Feature Suggestions
    • Climb the leaderboard!
    • Swag Shop
    • Contact
    • Global Feedback Survey
  • Support
  • PARTNERS
    • All Partners
    • Technology Partners
    • Strategic Partner Program
    • Deal Registration
  • >> Go to dataminer.services

Good balance between number and size of data packages

Solved826 views17th November 2022inter element communication protocol
2
Mieke Dryepondt [SLC] [DevOps Advocate]3.65K 17th November 2022 0 Comments

I need to communicate between elements (remote set) and find it difficult to determine between the number of packages to send and the size of a package.

For example in extremes, I could send many 'small' packages or a few 'gigantic' packages to another element.

What would be the benchmark of the number of packages and what would be the limited size of a single package?

Doing 4000 sets (1 set is already a group of instructions) in a loop feels like a load peak that could be reduced, but I don't want to impact something else (or miss data) by sending a package that is too large.

Mieke Dryepondt [SLC] [DevOps Advocate] Selected answer as best 17th November 2022

1 Answer

  • Active
  • Voted
  • Newest
  • Oldest
1
Avatar photo
Jan Staelens [SLC] [DevOps Advocate]889 Posted 17th November 2022 0 Comments

There is no single answer to this question. As you mentioned we need to balance the load vs the used memory. Both are linked to the setup. There's not real benchmarks for this.

Best practice would be to keep the number of remote calls to a minimum and at the same time avoid an out of memory exception.

As this is all linked to the setup and of course your use case (what data needs to be passed, how frequent, and should all data be send right away or is a buffer mechanism work to spread the load)

All I would say is to default to a minim amount of calls as a best practice and only consider changing that if and when you detect it is a problem.

This question has a very high risk of overcomplicating or over-engineering your code which can lead to a less stable setup that's difficult to maintain. So try to remember KISS, Keep it Simple & Stupid.

If you are still interested in making something more tailered to different setups. For example if you know you are making a highly re-usable and dynamic module or setup. Then you could build a configurable mechanism to send the data in batches.

This way you can at run-time still adapt according to the needs of the setup. e.g. reduce the consumed memory by sending more packages and perhaps in a broader time frame.

There's even options to make a self-regulating setup, where you monitor your own system and have the code automatically change the package sizes based on measured CPU Spikes...

But again, this is in most scenario's a bad idea, and overcomplicating your code.

Mieke Dryepondt [SLC] [DevOps Advocate] Selected answer as best 17th November 2022
You are viewing 1 out of 1 answers, click here to view all answers.
Please login to be able to comment or post an answer.

My DevOps rank

DevOps Members get more insights on their profile page.

My user earnings

0 Dojo credits

Spend your credits in our swag shop.

0 Reputation points

Boost your reputation, climb the leaderboard.

Promo banner DataMiner DevOps Professiona Program
DataMiner Integration Studio (DIS)
Empower Katas
Privacy Policy • Terms & Conditions • Contact

© 2025 Skyline Communications. All rights reserved.

DOJO Q&A widget

Can't find what you need?

? Explore the Q&A DataMiner Docs

[ Placeholder content for popup link ] WordPress Download Manager - Best Download Management Plugin