Skip to content
DataMiner DoJo

More results...

Generic selectors
Exact matches only
Search in title
Search in content
Post Type Selectors
Search in posts
Search in pages
Search in posts
Search in pages
Log in
Menu
  • Blog
  • Questions
  • Learning
    • E-learning Courses
    • Open Classroom Training
    • Certification
      • DataMiner Fundamentals
      • DataMiner Configurator
      • DataMiner Automation
      • Scripts & Connectors Developer: HTTP Basics
      • Scripts & Connectors Developer: SNMP Basics
      • Visual Overview – Level 1
      • Verify a certificate
    • Tutorials
    • Video Library
    • Books We Like
    • >> Go to DataMiner Docs
  • Expert Center
    • Solutions & Use Cases
      • Solutions
      • Use Case Library
    • Markets & Industries
      • Media production
      • Government & defense
      • Content distribution
      • Service providers
      • Partners
      • OSS/BSS
    • DataMiner Insights
      • Security
      • Integration Studio
      • System Architecture
      • DataMiner Releases & Updates
      • DataMiner Apps
    • Agile
      • Agile Webspace
      • Everything Agile
        • The Agile Manifesto
        • Best Practices
        • Retro Recipes
      • Methodologies
        • The Scrum Framework
        • Kanban
        • Extreme Programming
      • Roles
        • The Product Owner
        • The Agile Coach
        • The Quality & UX Coach (QX)
    • DataMiner DevOps Professional Program
  • Downloads
  • More
    • Feature Suggestions
    • Climb the leaderboard!
    • Swag Shop
    • Contact
      • General Inquiries
      • DataMiner DevOps Support
      • Commercial Requests
    • Global Feedback Survey
  • PARTNERS
    • All Partners
    • Technology Partners
    • Strategic Partner Program
    • Deal Registration
  • >> Go to dataminer.services

Cassandra DB: “grace seconds” default value and related custom tuning

Solved651 views12th November 2024adl2099 Cassandra DB gc_grace_seconds
0
Alberto De Luca [DevOps Enabler]4.58K 29th October 2024 0 Comments

Hello dojo,
Looking for some more info on the default value for gc_grace_seconds (on the data and elementdata tables, if different) in local Cassandra DBs and any trade-off when changing this value (what's recommended to handle tombstones efficiently and/or to avoid the need for frequent compactions)

I understand this can also vary with the specific DM version - depending on what is configured before upgrading the DMS:  https://community.dataminer.services/question/upgrade-behavior-for-gc_grace_seconds/

Thanks

Alberto De Luca [DevOps Enabler] Selected answer as best 12th November 2024

1 Answer

  • Active
  • Voted
  • Newest
  • Oldest
3
Laurens Vercruysse [SLC] [DevOps Advocate]346 Posted 7th November 2024 3 Comments

Hi Alberto,

The gc_grace_seconds value is designed to "exclude" a specific tombstoned (=deleted) record in Cassandra for the time specified in "gc_grace_seconds" from compaction.
This is mainly done to prevent these records from becoming "desynchronized" in case Cassandra nodes go down.
If a node goes down, it does not receive the instruction to write tombstones, but the rest of the Cassandra cluster, which has written the tombstones to disk, may still be up.
Would we immediately trigger compaction, effectively removing the tombstones from the rest of the cluster, we would see an effect where, when the node that was down comes up again, would still have the records in their non-tombstoned, non-deleted form. At that point, we have effectively created "zombie data".

The defaults as of present are 4 hours for items with a TTL, such as trending and a full day for items that do not expire by TTL, which is most of the other data, such as configuration data or alarms.

The defaults are a result of careful consideration. We considered e.g. what downtimes we encounter and what data benefits the most from being fully consistent and compared that to the amount of overhead on the disk the setting caused (i.e. the amount of additional disk space this takes).

Usually, we do not recommend deviating from these values.
Deviations are often only done when the system is has a very fast write/delete load, causing a lot of disk space to be taken up in a short time. Then, diminishing the gc_grace_seconds can potentially help as a temporary measure until the write/delete load can be diminished.

Note: for now, DataMiner sets back the default gc_grace_seconds upon each startup.
We are planning to allow more flexibility for that though.

Alberto De Luca [DevOps Enabler] Selected answer as best 12th November 2024
Alberto De Luca [DevOps Enabler] commented 12th November 2024

Thanks for the prompt feedback, Laurens

Are there any different considerations to be done for the scenario where each DMA has its own local Cassandra DB?

Asking in the context of agents with and without failover license: our goal would be to minimize the need for Cassandra compactions.

Laurens Vercruysse [SLC] [DevOps Advocate] commented 12th November 2024

Hi Alberto, on Windows, we toggle a manual "Major compaction" once per week for all the tables. This does not happen on Linux (i.e. Cassandra Cluster), where a "Minor compaction" is sufficient. The gc_grace remains the same on Cassandra single. As such, when a node has been down for a day or more, you may see some of that "zombie data". On Windows, the major compactions are unavoidable. Minor compactions are automatically triggered by Cassandra, depending on the compaction strategy used and a set of timers/triggers defined by Cassandra: https://cassandra.apache.org/doc/latest/cassandra/managing/operating/compaction/overview.html#types-of-compaction
As such, it is not possible to have strict control over the amount of compactions, save for the Major compactions that we schedule.

Alberto De Luca [DevOps Enabler] commented 12th November 2024

Thanks for your feedback, Lauren – we'll review with our squad – this was really helpful

Please login to be able to comment or post an answer.

My DevOps rank

DevOps Members get more insights on their profile page.

My user earnings

0 Dojo credits

Spend your credits in our swag shop.

0 Reputation points

Boost your reputation, climb the leaderboard.

Promo banner DataMiner DevOps Professiona Program
DataMiner Integration Studio (DIS)
Empower Katas

Recent questions

Correlation Engine: “Test rule” doesn’t result in a hit, despite functional rule 1 Answer | 3 Votes
When using the Setter = true attribute, will the copy action always be executed first? 1 Answer | 3 Votes
Multiple Set on Table parameters for DVE’s 1 Answer | 2 Votes

Question Tags

adl2099 (115) alarm (62) Alarm Console (82) alarms (100) alarm template (83) Automation (223) automation scipt (111) Automation script (167) backup (71) Cassandra (180) Connector (109) Correlation (69) Correlation rule (52) Cube (150) Dashboard (194) Dashboards (188) database (83) DataMiner Cube (57) DIS (81) DMS (71) DOM (140) driver (65) DVE (56) Elastic (83) Elasticsearch (115) elements (80) Failover (104) GQI (159) HTTP (76) IDP (74) LCA (152) low code app (166) low code apps (93) lowcodeapps (75) MySQL (53) protocol (203) QAction (83) security (88) SNMP (86) SRM (337) table (54) trending (87) upgrade (62) Visio (539) Visual Overview (345)
Privacy Policy • Terms & Conditions • Contact

© 2025 Skyline Communications. All rights reserved.

DOJO Q&A widget

Can't find what you need?

? Explore the Q&A DataMiner Docs

[ Placeholder content for popup link ] WordPress Download Manager - Best Download Management Plugin