Skip to content
DataMiner DoJo

More results...

Generic selectors
Exact matches only
Search in title
Search in content
Post Type Selectors
Search in posts
Search in pages
Search in posts
Search in pages
Log in
Menu
  • Blog
  • Questions
  • Learning
    • Courses
    • Certification
      • DataMiner Operator
      • DataMiner Administrator
      • DataMiner Automation
      • Scripts & Connectors Developer: HTTP Basics
      • Scripts & Connectors Developer: SNMP Basics
      • Visual Overview – Level 1
    • Video Library
    • Books We Like
    • Third-party Courses
    • DevOps Enablement Services
    • Quizzes
  • RESOURCES
  • Agile
    • Agile Webspace
    • Everything Agile
      • The Agile Manifesto
      • Best Practices
      • Retro Recipes
    • Methodologies
      • The Scrum Framework
      • Kanban
      • Extreme Programming
    • Roles
      • The Product Owner
      • The Agile Coach
      • The Quality & UX Coach (QX)
  • Expert Hubs
    • DataMiner DevOps Professional Program
    • DataMiner
      • DataMiner Releases & Updates
      • System Architecture
      • Integration Studio
      • Visual Overview
      • Security
    • Technologies
      • CI/CD
      • Converged Interconnect Networks
      • DOCSIS 3.1 / DAA
      • Elasticsearch
      • Kubernetes
      • Telemetry / Flow Monitoring
      • PTP
      • SDN / ST2110 / NMOS
    • Business
      • Software Business models
      • Perpetual Use or Usage-based Services
  • Discover
    • Markets
      • Media production
      • Government & defense
      • Content distribution
      • Service providers
      • Partners
      • OSS/BSS
    • Solutions
    • Use Case Library
    • DataMiner Dashboards
    • DataMiner Downloads
    • DataMiner Modules
      • Core features
      • SRM features
    • DataMiner Apps
    • Professional Services
  • PARTNERS
    • All Partners
    • Technology Partners
    • Strategic Partner Program
    • Deal Registration
  • Pricing
  • More
    • Feature Suggestions
    • Online DataMiner Systems
    • Leaderboard
    • Swag Shop
    • Contact
      • General Inquiries
      • DataMiner DevOps Support
      • Commercial Requests
    • Global Feedback Survey
    • Verify a certificate
  • >> Go to dataminer.services

Migration to DB Cluster ( Cassandra and Elastic) doubts

Solved149 views19th June 2023Cassandra DB database migration Elasticsearch
4
Bruno Sousa [DevOps Advocate]549 7th June 2023 0 Comments

Hello

We are currently migrating from the "per DMA" local databases to a "centralized cluster" databases approach, both for Cassandra and Elastic.

We need to have two databases instances in the Cassandra and Elastic DB centralized clusters though: one database for the pre-prod Dataminer instance and other for the production Dataminer instance.

We are starting with the migration of the pre-prod dataminer instance to these centralized clusters.

We followed the migration procedure and some doubts arose:

1- We have multiple keyspaces (one per table) in the Cassandra cenralized cluster after the migration. Is supposed to be this way?

2- We don't have alarm tables in the Cassandra centralized cluster. Is supposed to be this way? All alarms will only be stored in the Elastic Search centralized instance?

3- From now on, all the data (both trend and alarm) will only be kept in the centralized cluster and not on the local DMA databases right?

Thank you.

Best regards

Bruno Sousa

Bruno Sousa [DevOps Advocate] Selected answer as best 19th June 2023

1 Answer

  • Active
  • Voted
  • Newest
  • Oldest
3
Michiel Saelen [SLC] [DevOps Enabler]3.91K Posted 19th June 2023 1 Comment

Hi Bruno,

1 - Yes they created more keyspaces for the tables, this is to have more freedom to configure what replication strategy/factor you want to apply for every set of data. We did get complaints about the amount of keyspaces to manage, so we are looking to combine again a couple of tables into a known set of keyspaces. This will make it easier to manage the user permissions on the keyspaces if you do this.

2 - Yes, Alarms have been migrated towards Elastic. The reason for this is that we have to retrieve alarms in many different ways (e.g. based on view level, element level, service level, ...) in order to do this efficiently we stored the alarms multiple times in Cassandra with different keys for retrieval. This was causing some overhead and for systems with a lot of alarms (and big alarms), we saw a heavy load on Cassandra because of this. So the decision was made to move it to Elastic which allows us to use the indexing capabilities from Elastic instead of storing it multiple times.

3 - Once your data is migrated from the local DB to the central DB, the local DB is no longer needed. And as you probably know already the centralized DB has many benefits scaling independently, swarming capabilities in upcoming releases, ...

Bruno Sousa [DevOps Advocate] Posted new comment 19th June 2023
Bruno Sousa [DevOps Advocate] commented 19th June 2023

Hi Michiel,

Thank you for the clarification!

Best regards
Bruno Sousa

Please login to be able to comment or post an answer.

My Dojo Points

0 Credits
0 Reputation

Question Tag Cloud

adl2099 (54) Alarm Console (53) alarms (70) alarm template (58) API (35) Automation (144) automation scipt (65) Automation script (95) backup (54) Cassandra (142) cloud (37) Connector (44) Correlation (42) Cube (114) Dashboard (134) Dashboards (136) database (64) DataMiner Cube (37) DIS (53) DMS (48) DOM (68) driver (48) DVE (43) Elastic (73) Elasticsearch (99) elements (49) Failover (90) GQI (51) HTTP (45) IDP (55) low code app (70) low code apps (43) MySQL (50) protocol (145) QAction (58) resources (35) security (71) SNMP (56) SRM (267) SRM resources (39) timeout (35) trending (66) upgrade (44) Visio (373) Visual Overview (290)
Privacy Policy • Terms & Conditions • Contact

© 2023 Skyline Communications. All rights reserved.

[ Placeholder content for popup link ] WordPress Download Manager - Best Download Management Plugin