Skip to content
DataMiner DoJo

More results...

Generic selectors
Exact matches only
Search in title
Search in content
Post Type Selectors
Search in posts
Search in pages
Search in posts
Search in pages
Log in
Menu
  • Blog
  • Questions
  • Learning
    • E-learning Courses
    • Open Classroom Training
    • Certification
      • DataMiner Fundamentals
      • DataMiner Configurator
      • DataMiner Automation
      • Scripts & Connectors Developer: HTTP Basics
      • Scripts & Connectors Developer: SNMP Basics
      • Visual Overview – Level 1
      • Verify a certificate
    • Tutorials
    • Video Library
    • Books We Like
    • >> Go to DataMiner Docs
  • Expert Center
    • Solutions & Use Cases
      • Solutions
      • Use Case Library
    • Markets & Industries
      • Media production
      • Government & defense
      • Content distribution
      • Service providers
      • Partners
      • OSS/BSS
    • DataMiner Insights
      • Security
      • Integration Studio
      • System Architecture
      • DataMiner Releases & Updates
      • DataMiner Apps
    • Agile
      • Agile Webspace
      • Everything Agile
        • The Agile Manifesto
        • Best Practices
        • Retro Recipes
      • Methodologies
        • The Scrum Framework
        • Kanban
        • Extreme Programming
      • Roles
        • The Product Owner
        • The Agile Coach
        • The Quality & UX Coach (QX)
    • DataMiner DevOps Professional Program
  • Downloads
  • More
    • Feature Suggestions
    • Climb the leaderboard!
    • Swag Shop
    • Contact
      • General Inquiries
      • DataMiner DevOps Support
      • Commercial Requests
    • Global Feedback Survey
  • PARTNERS
    • All Partners
    • Technology Partners
    • Strategic Partner Program
    • Deal Registration
  • >> Go to dataminer.services

Cassandra Cluster large partitions relating to Skyline Queue Manager elements

Solved475 views5th March 2024cassandra cluster partitions
3
Arunkrishna Shreeder [SLC] [DevOps Advocate]4.00K 1st March 2024 0 Comments

Hi all,

We have these 2 elements in our DMA with large partitions :

The Skyline Queue Manager elements are used by our provisioning solution to pass tokens and associated data from one process to the next. From visually observing the Queue elements, they do not display large tables. It is noticed that during the provisioning process, there can be many tokens being generated. The tokens are displayed in the Queue element’s Incoming Tokens queue which then moves to the Outgoing table then the Completed table. But after the provisioning process, these tables are empty as they are processed. So

1. What data is actually being held within these partitions that correspond to the queue elements and are they required to persist across different provisioning runs?

2. What is the impact of these elements having large partitions and what can be done to reduce the size of these partitions?

Thank you in advance.

Marieke Goethals [SLC] [DevOps Catalyst] Selected answer as best 5th March 2024

1 Answer

  • Active
  • Voted
  • Newest
  • Oldest
3
Michiel Saelen [SLC] [DevOps Enabler]5.63K Posted 4th March 2024 0 Comments

I had a quick look at this connector. It is used to keep track of tokens in PA. The 1.0.1.X range is saving all (incoming, outgoing, completed) tokens in tables. When a table has saved columns (PK and DK columns are saved by default unless you use the volatile option), the data is pushed to DB. So that when an element restarts the data is still there. This data is stored in Cassandra. Without going too deep into the details, Cassandra will keep track of all manipulations on the table, so it could be that the partition is larger than the actual data. This is to ensure if you have multiple nodes in Cassandra that everyone can keep track of what happened. (For example, if you send a delete of a row while one of the Cassandra nodes is down, you can tell it when it comes back up that data is removed at a later stage than when that node has it as an insert). So if you have a table in a connector that is not volatile and you have some columns that are saved that change a lot you can have potentially large partitions. This is something you should avoid as large partitions will result in a lot of mem/cpu usage.

This is why in the new range 1.0.2.X of the connector they removed those saved tables and they made a single volatile table with active tokens. So I would advise to see if you can upgrade the Process Automation solution to a later version that fixes this problem.

Marieke Goethals [SLC] [DevOps Catalyst] Selected answer as best 5th March 2024
You are viewing 1 out of 1 answers, click here to view all answers.
Please login to be able to comment or post an answer.

My DevOps rank

DevOps Members get more insights on their profile page.

My user earnings

0 Dojo credits

Spend your credits in our swag shop.

0 Reputation points

Boost your reputation, climb the leaderboard.

Promo banner DataMiner DevOps Professiona Program
DataMiner Integration Studio (DIS)
Empower Katas

Recent questions

Web Applications exception in Cube due to invalid certificate 0 Answers | 0 Votes
Redundancy Groups and Alarming – Duplicate Alarms 0 Answers | 0 Votes
Correlation Engine: “Test rule” doesn’t result in a hit, despite functional rule 1 Answer | 3 Votes

Question Tags

adl2099 (115) alarm (62) Alarm Console (82) alarms (100) alarm template (83) Automation (223) automation scipt (111) Automation script (167) backup (71) Cassandra (180) Connector (109) Correlation (69) Correlation rule (52) Cube (151) Dashboard (194) Dashboards (188) database (83) DataMiner Cube (57) DIS (81) DMS (71) DOM (140) driver (65) DVE (56) Elastic (83) Elasticsearch (115) elements (80) Failover (104) GQI (159) HTTP (76) IDP (74) LCA (152) low code app (166) low code apps (93) lowcodeapps (75) MySQL (53) protocol (203) QAction (83) security (88) SNMP (86) SRM (337) table (54) trending (87) upgrade (62) Visio (539) Visual Overview (345)
Privacy Policy • Terms & Conditions • Contact

© 2025 Skyline Communications. All rights reserved.

DOJO Q&A widget

Can't find what you need?

? Explore the Q&A DataMiner Docs

[ Placeholder content for popup link ] WordPress Download Manager - Best Download Management Plugin