Hi all,
Recently, I've heard quite a few cases where Cassandra started to misbehave and by analyzing memory dumps, we could determine the issues were caused by an excessive amount of tombstones.
My question is, can we clarify for sure which actions (mainly protocol calls?) will result in tombstones and how many tombstones?
- When deleting a row which has 5 saved columns, will this result in 1 tombstone or will this result in 5 tombstones? or... ?
- When clearing a cell of a saved column using protocol.Clear, will this result in 1 or more tombstones?
- When updating the value of a saved parameter, will this result in any tombstone?
- Can you think of any situation, other than the ones mentioned above, which could result in any tombstones?
Regards,
Simon Vandamme [SLC] [DevOps Advocate] Selected answer as best 27th August 2024
Do you happen to know if this is less common on external (linux based) Cassandra nodes rather than on Windows-based nodes used as the local DB?
By default, the Primary key, Display Key (and all columns that make up the display key) are saved. So I'll assume you included those when you say 5 saved columns.
Tombstones on Cassandra side are created in the following cases:
- DELETE command
- Expired data (TTL)
- INSERT or UPDATE command with a null value
- Inserting data into parts of a collection
- When deleting a row with 5 saved columns (e.g. PK and 4 other columns that have save option) you will create 5 tombstones (on row level in Cassandra as every saved table cell is a row in the elementdata table in Cassandra)
- This one I'm not a 100% sure and I would need to verify with SW, but if we write a null towards the table cells in Cassandra we potentially have 2 tombstones for every clear on a cell/parameter (on cell level in Cassandra on two columns v and vu)
- This should not result in a tombstone on the elementdata table
- It all depends what DataMiner will send towards the DB layer when doing certain commands (e.g. fillarray, setparameters, ...). If you want to verify how your tombstones increase you can connect with CQLSH on the linux machine and run the command 'select * from "SLDMADB".elementdata where d = 514 and e = 115;' with tracing enabled for your element. This will give you a good understanding how many tombstones are retrieved (see screenshot).
Simon Vandamme [SLC] [DevOps Advocate] Selected answer as best 27th August 2024
FYI: you can also choose the easy way and avoid all of this by choosing STaaS 🙂 https://docs.dataminer.services/user-guide/Advanced_Functionality/Databases/STaaS/STaaS.html