Is it possible to offload the DataMiner logs to Graylog, a centralized log management platform? And if so, how can this be done?
In general there are a multitude of methods and capabilities available in DataMiner to ensure that the data from within a DataMiner System can be made available for other systems. This includes for example SNMP trap forwarding, a TCP socket for alarm messages, data offloading towards a customer-owned database (supporting MySQL, Oracle, MS SQLServer), a variety of north bound APIs, etc. Those are all well documented.
The first question for this kind of cases in general is always who’s going to integrate with whom. If the integration responsibility lays with the third party system, then the third party system can simply use any of the aforementioned methods (or a combination thereof) to access event data and performance data in DataMiner, and ingest it. Of course the question is what kind of data ingest capabilities & flexibility does that third party system have (i.e. those are typically far more limited than DataMiner when it comes to data ingest 😉). The other option is to provide a bespoke DataMiner integration to further accommodate the third party system, and work around the constraints that it may have.
The Graylog platform seems to be more oriented towards the ingest of log files. In that case, you could consider ingesting the offload files that DataMiner generates for the central database option. This would require you to create a ‘fake’ central database in the current versions of DataMiner to trigger DataMiner to generate those offload log files (one of the upcoming versions of DataMiner will support generating the offload log files without having to activate the central database option). These are plain CSV files, and the content is composed of human readable strings (i.e. not internal ID references as used by DataMiner). The files can be made accessible for the Graylog platform via a file or FTP share for example.
Note however that DataMiner will generate those log files, but it will not clean them up (as it doesn’t know if the third party system has picked those up or not). It would be the responsibility of the Graylog system to pick up those files and to remove those after that’s done. If it doesn’t do that, you would be at risk of filling up the disk space and locking up systems (because you cannot log events and performance data endlessly). And note that they have to pick those up from each individual DMA in the DMS. If the Graylog system does not have those capabilities, then alternatively a DataMiner driver could be used to do the housekeeping, and potentially even also to push the files to the third party system if needed (i.e. as an alternative to making them available via a directory or FTP share, and/or to consolidate those into one location). Similar set-ups have been used for offload into other third party platforms.
Another alternative would be to set up a central database, e.g. MySQL, and use the DataMiner offload capability (in which case DataMiner will also generate the offload log files, but it will also take the responsibility to move those from all DMAs into one central database and to clean up the temporary offload log files), and have the Graylog platform take it from the central SQL database. Of course, again, this under the assumption that they support that kind of capability.
We have seen this use-case before where DataMiner needs to offload data to systems which seem similar to the out of the box available central database feature. (e.g. GrayLog, Kinesis Firehose, custom database schemes, ...)
How to tackle:
- Make sure DataMiner generates the offload files
- Have a driver or external software that processes these files and pushes them into the 3rd party system of choice (in your case GrayLog).
The caveat:
- In DataMiner versions prior to 10.0.10, it is not possible to have these files generated without a hack. The 'hack' was configuring a central database with an unreachable destination. That way the files were never deleted by DataMiner upon offload, and it allowed a driver to process these.
- From DataMiner 10.0.10 onwards there will be a clean option to generate these offload files to have a supported, future proof implementation.