Dear Community,
We recently have encountered some sever alarm storms which lead to the elasticsearch cluster being flooded with alarm data. As a result from that, the cluster will run out of available space within the next 1 - 2 weeks.
To mitigate the situation on short term, we would like to purge all alarms older than 6 months from the elasticsearch cluster and are happy to perform this action manually through a request sent from postman.
Is it possible to do this with a single post request directly to the alias which combines all individual alarm indices? Would someone be able to provide some assistance on which endpoint + JSON body best can be used to achieve this result?
With a bit of trial and error I managed to get to the following queries which i executed through postman on the elasticsearch cluster:
search query for all alarms older than 6 months:
endpoint: GET http://[ES NODE IP]:9200/dms-alarms/_search
body:
{
"query": {
"bool" : {
"must" : [
{
"range": {
"CreationTime": {"lte" : "2022-04-15" }}
}
]}
}
}
delete by query for alarms older than 6 months:
endpoint: POST http://[ES NODE IP]:9200/dms-alarms/_delete_by_query
body:
{
"query": {
"bool" : {
"must" : [
{
"range": {
"CreationTime": {"lte" : "2022-04-15" }}
}
]}
}
}
Executing the query took approximately 40 minutes. It covered about 10 million documents and managed to delete them without any issue.
exactly the same query can be executed on the information events indices by replacing dms-alarms with dms-info
Example: POST http://[ES NODE IP]:9200/dms-info/_delete_by_query