I am also interested in possibilities DataMiner has retrieving & storing NetFlow-Data. Also keen to learn about the performance (number of netflow records/s) and how NetFlow data gets stored in DataMiner.
I can confirm that we currently have the capability of capturing, storing and presenting informative dashboards on the recorded netflow traffic. We capture the data by having a Netflow collector element installed and operating on a DataMiner agent. This element will parse the incoming netflow traffic and complete it with extra metadata such as a hostname to IP mapping and Application name to IP port mapping information.
It also features the capabilities of optionally doing Top flow filtering (basically filter out the many tiny data communication packets) and to perform data aggregation to allow high resolution data captures to be stored for a small timeframe whilst storing lower resolution data captures for a longer timeperiod.
We store this processed netflow traffic in a document store database called ElasticSearch. This will allow us to be very flexible on the reporting side of the captured netflow traffic.
Typically a single DMA can process up to 1500 - 2000 incoming netflow packets per second which translate into approx. 35000 - 45000 netflow flows per second. We can horizontally scale this processing by installing more DataMiner agents each hosting a netflow collector element. At the same time, the Elasticsearch cluster where the netflow traffic will be stored needs to be scaled accordingly as well.
Once the data is available in the Elasticsearch cluster, we offer a dedicated easy to configure dashboard component which allows you to:
- Select which data you want to see.
- How this data needs to be grouped and aggregated.
- How this data needs to be presented.
Great info Jeroen. And we have more coming… I’m looking forward to see some of this Netflow data being leveraged in the new node-edge component, and the real kicker will come when this is combined / overlayed with data from other sources (e.g. via the Generic Query Interface). Quick follow-up question, as the data is stored in a LoggerTable (I presume, because you mention the records going into ElasticSearch), are those records then also available in the Search page in the DataDisplay of the collectors (where you can do key word search queries)?