NetFlow – How long is data kept in ElasticSearch and why is it spread across different tables?

Solved842 viewsElasticsearch logger table NetFlow
0
0 Comments

Hi Dojo,

I have a Netflow data collector running for 2 years now.
Looking at the ElasticSearch indices, I noticed that some logger tables (1000 and 2000) have multiple indices. What's the reason for that?

green open dms-dynamic_elementdata_17_63164_2000-2020.08.17.12-000001 > 5.7gb
green open dms-dynamic_elementdata_17_63164_3000-2020.08.17.12-000001 > 4.7gb
yellow open dms-dynamic_elementdata_17_63164_2000-2021.11.05.13-000002 > 1.1gb
yellow open dms-dynamic_elementdata_17_63164_1000-2021.11.05.13-000002 > 1.5gb
green open dms-dynamic_elementdata_17_63164_1000-2020.08.17.12-000001 > 7.2gb

I also want to investigate how long the data is stored.
Is there a way that I can see the oldest entries? Can I configure how long the data is kept?

Marieke Goethals [SLC] [DevOps Catalyst] Selected answer as best 17th July 2023

1 Answer

1
1.60K 1 Comment

Hi Jens,

A partial answer to your questions. To see the oldest entries, you can use the assigned alias to the indices created by the logger table:

GET dms-dynamic_elementdata_162101_623_2000/_search
{
"query": {
"match_all": {}
},
"sort": [
{
"SL_Internal_TimeField": {
"order": "asc"
}
}
] }

By default, a total of 10 hits are returned, but you can use the size query parameter (GET dms-dynamic_elementdata_162101_623_2000/_search?size=20) to get additional documents or the Scroll API to iterate over all the documents in a batch of different sizes.

Marieke Goethals [SLC] [DevOps Catalyst] Selected answer as best 17th July 2023
Gelber Mahecha [SLC] [DevOps Enabler] commented

Hi Jans,

Another partial answer: In a recent case, I had the opportunity to confirm that we can use the Partition tag on column definition with DATETIME and DataMiner will clean the documents older than the specified time window.

For instance, a table’s column with the following definition will keep documents for the last year based on the time values stored in this field:

ColumnDefinition = DATETIME
Partition partitionsToKeep=”12″ = month

As per our development documentation, the column should be filled up with a specific DateTime format (though that part applied to the Cassandra log tables).