We are currently migrating from the "per DMA" local databases to a "centralized cluster" databases approach, both for Cassandra and Elastic.
We need to have two databases instances in the Cassandra and Elastic DB centralized clusters though: one database for the pre-prod Dataminer instance and other for the production Dataminer instance.
We are starting with the migration of the pre-prod dataminer instance to these centralized clusters.
We followed the migration procedure and some doubts arose:
1- We have multiple keyspaces (one per table) in the Cassandra cenralized cluster after the migration. Is supposed to be this way?
2- We don't have alarm tables in the Cassandra centralized cluster. Is supposed to be this way? All alarms will only be stored in the Elastic Search centralized instance?
3- From now on, all the data (both trend and alarm) will only be kept in the centralized cluster and not on the local DMA databases right?
1 - Yes they created more keyspaces for the tables, this is to have more freedom to configure what replication strategy/factor you want to apply for every set of data. We did get complaints about the amount of keyspaces to manage, so we are looking to combine again a couple of tables into a known set of keyspaces. This will make it easier to manage the user permissions on the keyspaces if you do this.
2 - Yes, Alarms have been migrated towards Elastic. The reason for this is that we have to retrieve alarms in many different ways (e.g. based on view level, element level, service level, ...) in order to do this efficiently we stored the alarms multiple times in Cassandra with different keys for retrieval. This was causing some overhead and for systems with a lot of alarms (and big alarms), we saw a heavy load on Cassandra because of this. So the decision was made to move it to Elastic which allows us to use the indexing capabilities from Elastic instead of storing it multiple times.
3 - Once your data is migrated from the local DB to the central DB, the local DB is no longer needed. And as you probably know already the centralized DB has many benefits scaling independently, swarming capabilities in upcoming releases, ...