Skip to content
DataMiner DoJo

More results...

Generic selectors
Exact matches only
Search in title
Search in content
Post Type Selectors
Search in posts
Search in pages
Search in posts
Search in pages
Log in
Menu
  • Blog
  • Questions
  • Learning
    • E-learning Courses
    • Open Classroom Training
    • Certification
      • DataMiner Fundamentals
      • DataMiner Configurator
      • DataMiner Automation
      • Scripts & Connectors Developer: HTTP Basics
      • Scripts & Connectors Developer: SNMP Basics
      • Visual Overview – Level 1
      • Verify a certificate
    • Tutorials
    • Video Library
    • Books We Like
    • >> Go to DataMiner Docs
  • Expert Center
    • Solutions & Use Cases
      • Solutions
      • Use Case Library
    • Markets & Industries
      • Media production
      • Government & defense
      • Content distribution
      • Service providers
      • Partners
      • OSS/BSS
    • DataMiner Insights
      • Security
      • Integration Studio
      • System Architecture
      • DataMiner Releases & Updates
      • DataMiner Apps
    • Agile
      • Agile Webspace
      • Everything Agile
        • The Agile Manifesto
        • Best Practices
        • Retro Recipes
      • Methodologies
        • The Scrum Framework
        • Kanban
        • Extreme Programming
      • Roles
        • The Product Owner
        • The Agile Coach
        • The Quality & UX Coach (QX)
    • DataMiner DevOps Professional Program
  • Downloads
  • More
    • Feature Suggestions
    • Climb the leaderboard!
    • Swag Shop
    • Contact
      • General Inquiries
      • DataMiner DevOps Support
      • Commercial Requests
    • Global Feedback Survey
  • PARTNERS
    • All Partners
    • Technology Partners
    • Strategic Partner Program
    • Deal Registration
  • >> Go to dataminer.services

Is it possible to offload DataMiner data and log files to Graylog?

Solved2.07K views7th September 2020Graylog offload
3
pramod Gupta [SLC] [DevOps Member]255 22nd July 2020 0 Comments

Is it possible to offload the DataMiner logs to Graylog, a centralized log management platform? And if so, how can this be done?

pramod Gupta [SLC] [DevOps Member] Selected answer as best 7th September 2020

2 Answers

  • Active
  • Voted
  • Newest
  • Oldest
3
Ben Vandenberghe [SLC] [DevOps Enabler]9.06K Posted 22nd July 2020 0 Comments

In general there are a multitude of methods and capabilities available in DataMiner to ensure that the data from within a DataMiner System can be made available for other systems.  This includes for example SNMP trap forwarding, a TCP socket for alarm messages, data offloading towards a customer-owned database (supporting MySQL, Oracle, MS SQLServer), a variety of north bound APIs, etc.  Those are all well documented.

The first question for this kind of cases in general is always who’s going to integrate with whom.  If the integration responsibility lays with the third party system, then the third party system can simply use any of the aforementioned methods (or a combination thereof) to access event data and performance data in DataMiner, and ingest it.   Of course the question is what kind of data ingest capabilities & flexibility does that third party system have (i.e. those are typically far more limited than DataMiner when it comes to data ingest 😉).  The other option is to provide a bespoke DataMiner integration to further accommodate the third party system, and work around the constraints that it may have.

The Graylog platform seems to be more oriented towards the ingest of log files.  In that case, you could consider ingesting the offload files that DataMiner generates for the central database option.   This would require you to create a ‘fake’ central database in the current versions of DataMiner to trigger DataMiner to generate those offload log files (one of the upcoming versions of DataMiner will support generating the offload log files without having to activate the central database option).  These are plain CSV files, and the content is composed of human readable strings (i.e. not internal ID references as used by DataMiner).   The files can be made accessible for the Graylog platform via a file or FTP share for example.

Note however that DataMiner will generate those log files, but it will not clean them up (as it doesn’t know if the third party system has picked those up or not).  It would be the responsibility of the Graylog system to pick up those files and to remove those after that’s done.   If it doesn’t do that, you would be at risk of filling up the disk space and locking up systems (because you cannot log events and performance data endlessly).  And note that they have to pick those up from each individual DMA in the DMS.  If the Graylog system does not have those capabilities, then alternatively a DataMiner driver could be used to do the housekeeping, and potentially even also to push the files to the third party system if needed (i.e. as an alternative to making them available via a directory or FTP share, and/or to consolidate those into one location).  Similar set-ups have been used for offload into other third party platforms.

Another alternative would be to set up a central database, e.g. MySQL, and use the DataMiner offload capability (in which case DataMiner will also generate the offload log files, but it will also take the responsibility to move those from all DMAs into one central database and to clean up the temporary offload log files), and have the Graylog platform take it from the central SQL database.  Of course, again, this under the assumption that they support that kind of capability.

pramod Gupta [SLC] [DevOps Member] Selected answer as best 7th September 2020
7
Jan Vanhove [SLC] [DevOps Member]1.16K Posted 22nd July 2020 0 Comments

We have seen this use-case before where DataMiner needs to offload data to systems which seem similar to the out of the box available central database feature.  (e.g. GrayLog, Kinesis Firehose, custom database schemes, ...)

How to tackle:

  • Make sure DataMiner generates the offload files
  • Have a driver or external software that processes these files and pushes them into the 3rd party system of choice (in your case GrayLog).

The caveat:

  • In DataMiner versions prior to 10.0.10, it is not possible to have these files generated without a hack.  The 'hack' was configuring a central database with an unreachable destination.  That way the files were never deleted by DataMiner upon offload, and it allowed a driver to process these.
  • From DataMiner 10.0.10 onwards there will be a clean option to generate these offload files to have a supported, future proof implementation.
pramod Gupta [SLC] [DevOps Member] Unselected an answer 7th September 2020
Please login to be able to comment or post an answer.

My DevOps rank

DevOps Members get more insights on their profile page.

My user earnings

0 Dojo credits

Spend your credits in our swag shop.

0 Reputation points

Boost your reputation, climb the leaderboard.

Promo banner DataMiner DevOps Professiona Program
DataMiner Integration Studio (DIS)
Empower Katas

Recent questions

Alarm Dashboard PDF/CSV Export 0 Answers | 0 Votes
Is the Microsoft SharePoint Connector Still Usable 0 Answers | 0 Votes
Is the Microsoft SharePoint Connector Still Usable 0 Answers | 0 Votes

Question Tags

adl2099 (115) alarm (62) Alarm Console (82) alarms (100) alarm template (83) Automation (223) automation scipt (111) Automation script (167) backup (71) Cassandra (180) Connector (108) Correlation (68) Cube (150) Dashboard (194) Dashboards (188) database (83) DataMiner Cube (57) DIS (81) DMS (71) DOM (139) driver (65) DVE (55) Elastic (83) Elasticsearch (115) elements (80) Failover (104) GQI (159) HTTP (76) IDP (74) LCA (151) low code app (166) low code apps (93) lowcodeapps (75) MySQL (53) protocol (203) QAction (83) security (88) services (51) SNMP (86) SRM (337) table (54) trending (87) upgrade (62) Visio (539) Visual Overview (345)
Privacy Policy • Terms & Conditions • Contact

© 2025 Skyline Communications. All rights reserved.

DOJO Q&A widget

Can't find what you need?

? Explore the Q&A DataMiner Docs

[ Placeholder content for popup link ] WordPress Download Manager - Best Download Management Plugin