We have configured Cassandra to run on a separate D drive but noticed that during compaction, it also uses a lot of space of our C partition, where we have the OS and DataMiner. Because the C partition has less space, we wonder about any action to prevent this situation...
All ssTables (the DB data on disk) is stored on the separate drive. When Cassandra does compaction it tries to clean up and group ssTables for more efficient lookup. To do this it needs to go over the ssTables and start write the data to new ssTables on the separate disk.
When going over the data the Cassandra process wants to take a lot of data into memory. If you have large partitions it can lead to large memory consumption. This could become so large that there is not enough Memory (RAM) and at that point Windows provides the Cassandra application with page files (memory on disk) to prevent the application from crashing. Most likely this is why you are impact on C disk during the compaction (Cassandra is eating all resources from the OS). This could lead to other processes in the end being impacted (like the DataMiner services). There are a couple of things that you can do to improve this:
- Evaluate if you can reduce DB load (most effective)
This can be done by creating less alarms or alarm updates. If they have a lot of services make sure the service impact of your alarms is limited. Less trending, especially real time trending is heavy for DB as it needs to clean up a lot of data on every compaction cycle. This will heavily depend on your use case and what is possible for the customer. - Evaluate your page file settings
Could be that by enabling page files on more than only the C drive you are seeing less problems.
Note: you should always leave the page file settings on automatic to prevent crashes when limits are reached. - Move Cassandra to separate server
To prevent impact on DataMiner services you can move Cassandra to another server.
In case of virtualized environments make sure your disks are not shared or you could end up with the same problems again.
Thanks Michiel. At the first opportunity, we will check the configuration and sizes of the DB, to confirm if the page file is causing such usage on C partition.