Hi everyone,
We have a Dataminer cluster with 3 agents of 500 elements each, where we currently have 1100 elements. Each server has its own Cassandra database on one of its disks. However, since memory consumption increases to almost 100% during the compaction process, we want to migrate the databases to independent servers and add OpenSearch servers.
The question is: Is it sufficient to consider 3 Cassandra servers and 3 Elasticsearch servers, or what number of servers would you recommend for this cluster?
Resources:
Cassandra servers: RAM 64GB // Disk 1: 250GB SSD
Disk 2: 500GB SSD
OpenSearch server: RAM 64GB // Disk 1: 1TB SSD
Many thanks
Hi Isabel,
Looking at the data you provided, it looks like 3 servers for Cassandra and OpenSearch should be sufficient. Today you are able to run on 3 Cassandra nodes, although they max out during compaction, but in the new architecture, you are going to benefit from dedicated servers for the storage, and you will also have some load going from Cassandra to OpenSearch. Three servers also give you the possibility to make sure the cluster has some resilience if one node goes down. So, yes, I believe your plan looks great.
PS: If you want to take it one step further, you could also consider moving the storage to STaaS. Then your DMS would connect to cloud native storage from Microsoft Azure (Cosmos DB and Table Storage) managed by Skyline in dataminer.services. Then you only need to do one click to connect to STaaS and really forget about storage. No longer needed to build databases yourself, maintain and upgrade them, migrate to newer versions, patching, etc... The robustness, reliability and security levels you have now and forever in STaaS is very difficult to achieve yourselves with Cassandra and OpenSearch nodes on-prem. I believe it's better to put your valuable time in building solutions on top of DataMiner which will empower your team to be better and faster. Storage is just storage. It is readily available in STaaS, better, faster and cheaper... More info: Storage as a Service (STaaS) | DataMiner Docs
Bert
Thank you very much for your kind reply, Bert. We will also review more documentation regarding the STaaS solution.