solution Use Case
Kafka Consumer and Producer
Challenges of growing data
In today’s data-driven world, consolidating information from multiple sources is a common challenge. As companies grow, both the demand for information across various departments and the creation of new data sources increase. This often leads to repeated requests to the same data sources across different applications, causing inefficiencies, increased latency, data bottlenecks, and higher infrastructure costs. As companies expand, these problems multiply.
Develop a business data strategy with Kafka and DataMiner
By leveraging Apache Kafka’s scalability and DataMiner’s real-time monitoring and orchestration capabilities, companies can significantly enhance their data management processes, reducing redundancy and improving overall efficiency. DataMiner simplifies working with Kafka through its Generic Kafka Producer and Generic Kafka Consumer solutions:
- Generic Kafka Producer: As a producer client application, DataMiner collects data from various data sources, normalizes it, and sends it to Kafka streams. This makes the data accessible to any consuming application—whether within the DataMiner ecosystem or external systems—enabling seamless data distribution across platforms.
- Generic Kafka Consumer: DataMiner retrieves data from Kafka streams for processing, normalization, and visualization in modules like Dashboards and Low-Code Apps.
USE CASE DETAILS
DataMiner seamlessly polls information from any device, such as a GPON OLT, and forwards thousands of data points from millions of devices (ONTs) to Apache Kafka using the Kafka Producer. This integration ensures the data is readily available to any application within the ecosystem. As long as the Producer cycle is up and running, devices can be added or removed without issue.
DataMiner then retrieves the large amount of data from the Kafka stream using the Kafka Consumer, further normalizing them for analysis and monitoring. Historical data points are also preserved. Metrics are displayed to users through the Visual Overview in Cube.
These data points can also be shown in the DataMiner Dashboards app, which allows users to access the data even when they're out in the field.
With real-time data points available in DataMiner, a wide range of aggregations and calculations can be performed to derive high-level overviews that DataMiner is known for.