Our setup consists of a server running a standalone DataMiner agent (10.1.2), with a single Cassandra and Elasticsearch nodes.
For the past month, we've been getting a daily "Backup job had errors" notice.
The Backup log mentions "An error occurred when dumping the elastic database. Error: ".
The ElasticBackup log contains the following exception, 1 minute after the backup starts:
2021-09-10 00:01:19|Backup will be taken for following indices: dms-customdataindices, (............................), dms-ctickethistory-2021.02.17.07-000001
2021-09-10 00:02:19|System.NullReferenceException: Object reference not set to an instance of an object.
at SLDataGateway.Backup.Elastic.BackupManager.TakeSnapshot()
at SLDataGateway.Backup.Elastic.BackupManager.TakeBackup()
at SLDataGateway.Backup.Elastic.Program.Main(String[] args)
If I check the daily snapshot status using Kibana (query: GET /_snapshot/dms_backup/20210910), its state is "SUCCESS", with 0 failures.
The Elastic backup folder is also created daily, with what seems like all the correct contents.
How can I troubleshoot this further?
This has been solved since DataMiner 10.1.0 CU11/10.2.2 - the Elasticsearch backups taken from DataMiner/standalone tool had their timeout time increased from 1 to 15 min.
Hi Tiago,
I doubt the snapshot will contain all indices it lists in the logging.
From 10.1.4 onwards we have added null checks there that should help handle it beter and give a clear logging of the issue.
I would advise having a look in the Elasticsearch logging itself.
You can find it here C:\Program Files\Elasticsearch\logs.