Dear community,
we have a 3 node cassandra cluster with one Dataminer (the lab one) hooked on it. We monitor it by the Apache Cassandra Cluster Monitor connector. We have also installed the Cassandra Reaper that is periodically checking (and hopefully) fixing the tables.
And we see tombstones are growing for some tables. Mainly the tables of the non Dataminer keyspaces - reaper_db, system_auth, system. These are growing linearly like this one:
But there are also some tables of Dataminer keyspaces that seem to grow without any control:
Can you please advice what we could be missing in setup of our Cassandra cluster that would manage the tombstones count in a healthy limits?
Regards,
Milos
It is expected that the count will continue to increase as this is a counter (similar to packet counters in switches). The columns that might be interesting to understand if your tombstones keeps on rising is the Max and Percentiles columns. These will indicate what percentage of data was tombstones that were received when during reads towards the DB. As it is only registered on read it could be that you have many tombstones, but they only can be noticed when you perform a read. For example, for elementdata you will see a spike when restarting an element (that has a lot of table updates) as then DataMiner will read the data from DB to start up. The Max, Percentiles columns are for the last 5 minutes. FYI under the details section when double-clicking a cell or parameter you will find a description. In addition, from Cassandra 5.0 we will be able to read the logging which will be a better way to understand tombstone problems.
In short to know if you have a tombstone problem, look at the max trending for spikes to see if your spikes keep on raising.