Best practices for Cassandra Reaper schedules, gc_grace_seconds and tombstone management

58 views
4
0 Comments

Hi everyone,

I'm looking for some advice and best practices regarding Cassandra maintenance in a DataMiner environment.

We are currently running the following setup:

  • DataMiner 10.6 CU5
  • Apache Cassandra 4.1.10
  • 3-node Cassandra cluster on Ubuntu 24.04
  • Cassandra Reaper 3.3.4
  • Scheduled repairs every 7 days

Recently, we encountered an issue where a DataMiner element failed to initialize due to a database error (0x80040226).

After investigating, we found a high number of tombstones in the dmacluster_elementdata.elementdata table. Running a manual compaction on all three Cassandra nodes resolved the issue.

However, we monitored the table over the following days and noticed that the droppable tombstone ratio increased again:

Date
Droppable Tombstone Ratio

October 20 (after compaction)
0.12%

October 22
12.93%

October 23
16.94%

October 26
48.92%

The behavior is nearly identical across all three Cassandra nodes.

The table uses LeveledCompactionStrategy, and Cassandra is performing regular automatic compactions. Nevertheless, the tombstone ratio continues to increase.

During our investigation, we also discovered that Cassandra Reaper had been stopped for approximately eight days. We have restarted it, and the scheduled repairs have now completed successfully.

Questions for the community:

  1. Repair frequency vs. gc_grace_seconds

    The elementdata table has gc_grace_seconds = 86400 (24 hours), while our Reaper schedule runs every 7 days.

    Is this a recommended configuration for DataMiner, or should repairs be performed more frequently? Would increasing gc_grace_seconds be a better approach?

  2. Reaper configuration and table exclusions

    Are there recommended Reaper settings specifically for DataMiner environments?

    I understand that tables using TimeWindowCompactionStrategy (TWCS) should generally be excluded from automatic repairs. Is enabling blacklistTwcsTables: true sufficient, or are there other DataMiner-specific recommendations?

  3. Tombstone accumulation in elementdata

    Has anyone experienced similar tombstone accumulation in the elementdata table?

    Are there recommended adjustments to the compaction strategy or other ways to improve automatic tombstone cleanup without relying on periodic manual compactions?

  4. Reaper reliability

    How do others monitor Cassandra Reaper to ensure that scheduled repairs are actually running and completing successfully?

Our goal is to establish a reliable long-term maintenance configuration and avoid recurring element initialization failures.

I'd be interested to hear how others have configured Cassandra and Reaper in their DataMiner environments.

Thanks in advance for any suggestions or experiences!

Chris Glover [DevOps Advocate] Answered question

1 Answer

3
2.20K 0 Comments

You have my sympathy, we had a world of pain earlier this year due to excessive tombstones on the elementdata table which prevented elements restarting

  1. We have our gc_grace_seconds set to 172800 (2 days) on the elementdata table. Reaper is set to run every day.
  2. Our reaper looks at all tables, every 3 days, except element data which is reaped daily.
  3. Oh yes, we had an in-house created element that was doing thousands of writes and deletes to a table daily. This meant that the element would not restart as the tombstones were in the millions, thus causing a read timeout. USE VOLATILE TABLES WHERE YOU CAN!
  4. Use the cassandra cluster monitor element (https://catalog.dataminer.services/details/2ed38410-4e76-4081-b630-9c9d14f10db7). You will have to setup so access permissions on the tables. This has a repair History section so you can see when a table was repaired, most recent at the top.

in cassandra.yaml we addjusted the read  range timeoit value to 30s, it defaults to 10. This helps in high tombstone situations

read_request_timeout: 30000ms
range_request_timeout: 10000ms
write_request_timeout: 2000ms
counter_write_request_timeout: 5000ms
cas_contention_timeout: 1000ms
truncate_request_timeout: 60000ms
request_timeout: 10000ms
slow_query_log_timeout: 500ms
native_transport_timeout: 35000ms

There is a reason why we set the gc_grace_seconds to be longer than the reaper run interval, but it was a while ago and I can't remember what it was

You can run this to see howmany droppable tombstones there are per table

nodetool getsstables sldmadb_elementdata elementdata 400157:24 |xargs -I {} sstablemetadata {} | grep -i "droppable"

If you run nodetool repair abainst a table, or a specific elements entry, you need to run it on every node in your cluster. Likewise it's worth installing reaper on every node.

nodetool tablestats sldmadb_elementdata

Will show you the Top 10 elements with the highest tombstone count

If you have cqlsh, or DbVisualizer installed, you can run a very artificaly query such as

SELECT COUNT(*) FROM sldmadb_elementdata.elementdata WHERE d = 4 AND e = 24;

Where d is the DMA ID, and e is the element id. If it takes a long time to return that's the table with high tombstones.

Table compaction can help, or it can make it worse depending on which stratergy your using. As I said above, it's a world of pain!

Good luck!

Chris Glover [DevOps Advocate] Edited answer