Aggregate multiple source in one LCA

139 views
0
0 Comments

Hi all

I have to create a dashboard / LCA that should show different alarms and ticket from different sources, like other dataminer clusters, Grafana and Spark,

Now to integrate with Grafana I will try with the driver in the catalog Grafana Alerting API | Catalog (dataminer.services)

For Spark I forecast to create a Data Source Data Sources | DataMiner Docs

My doubts are about the best approch I can choose to gather the alarms from other Dataminer cluster, I see there is 3 different option

  • Custom API
  • Data Aggregator
  • Web Service

As I should expose all the alarms I get on a LCA or Dashboard, by keeping also the best low latency possibile, which should be the best solution for me? I think Custom API as it let me tailor the information I can get, but maybe there is a better solution I not see

Miguel Obregon [SLC] [DevOps Catalyst] Answered question 7th October 2024

1 Answer

0
16.30K 1 Comment

Hi Marco,

Regarding your question, I will first check if web services fit your requirements. If you see that you will need to implement multiple methods or process large chunks of information in your application, then I would recommend considering using a custom API.

Keep in mind that every time that you will open a dashboard, the queries defined in your dashboard/LCA will be triggered. This can impact performance depending on the complexity and volume of data being processed.

Data Aggregator can be used instead of custom APIs when you will need to process a high volume of data. For example, if you want to get the average value from the last 24 hours for a pool of KPIs available in multiple elements in your cluster, this query could take several minutes to complete depending of the number of elements involved.

In this scenario, your custom API would need to wait for a response from the cluster before displaying anything on your dashboard. This could lead to timeouts and degrade user experience.

For such use cases, I would recommend considering the Data Aggregator feature. This allows you to query information in advance so that results are readily available when you open the dashboard. The main drawback here is that you may not have the latest updates from the cluster at the time of querying.

Hope it helps.

Marco Di Blasi Posted new comment 7th October 2024
Marco Di Blasi commented

Hi Miguel
As one of the request is to keep the time to get the the last update the lowest possibile, at this point I’ll try with the Web Service / Custom API at the least to get the active alarms / last KPI.
I’ll try anyway to shrink how much data will be displayed to reduce the time to get the info from other cluster