We are evaluating DataMiner Data offloading for trend/config data to a central data warehouse.
Currently, the Data offload appears to all trended data. In our environment. Only a subset of parameters from specific protocols provide business value in the warehouse, while the remainder consumes unnecessary storage, bandwidth, and processing resources and will be immediately dropped.
Is there currently a mechanism to:
- Filter offloaded data by Protocol ID or Protocol Name?
- Filter by Parameter ID(s)?
- Filter by Element, Element Type, or Service?
- Configure inclusion/exclusion rules before data is forwarded to the Data Offload?
- Apply custom filtering logic at the collection layer rather than after ingestion into the warehouse?
We are looking for functionality similar to the event and alarm forwarding modules, where forwarding rules can be configured so only relevant data is sent from protocols/paramters
If this is not currently supported, has anyone implemented a scalable alternative, or is there an enhancement request already tracking this capability?
Hi Simon,
I believe there are two settings that can help reduce the volume of data being offloaded:
- Snapshots: Using information templates, you can determine which parameters are included in the offload (see Part of snapshot). In the database offload settings, the
Parameter Valueoption must be enabled. - Disabling offload at element level: You can modify the element's
Element.xmlfile to disable database offloading for that specific element.
Hope it helps.
Thanks for the feedback Miguel. We will test those options. Only issue with the second one is that we are a DaaS customer so do not have the option to modify the XML. If this was brought forward to the advanced element settings it would be more useful and scalable reducing the impact on the Skyline account team to update.