Connect external Elasticsearch with DMA

Solved1.28K views
1
0 Comments

Hi,

I have an issue when try to connect to external elasticsearch 6.8 with DMA, I’m using this DBConfiguration.xml config.

<?xml version=”1.0″?>
<DataBases xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xmlns:xsd=”http://www.w3.org/2001/XMLSchema” xmlns=”http://www.skyline.be/config/db”>
<DataBase active=”true” local=”true” type=”Elasticsearch”>
<ConnectString/>
<DBServer>35.175.118.112</DBServer>
<DSN/>
<DB></DB>
<UID>elastic</UID>
<PWD>changeme</PWD>
</DataBase>
</DataBases>

I get this error when restart DMA.

2021/07/31 16:40:33.765|SLLog|ApplyLogLevels|CRU|-1|Log Levels: Info: 0; Error: 0; Debug: 0 (from LogSettings.xml)
2021/07/31 16:40:33.769|SLDBConnection|SLDBConnection|INF|0|1|New DBGateway COM object created
2021/07/31 16:40:33.840|SLDBConnection|SLDBConnection|INF|0|1|Setting up connection to SLNet (Initial connection)
2021/07/31 16:40:34.407|SLDBConnection|SLDBConnection|INF|0|1|SLNet Connection has been established (da68abf6-7424-44c9-ace3-9482cab67ef8)
2021/07/31 16:40:34.583|SLDBConnection|DBGateway.Initialize|DBG|0|1|Loading configuration data…
2021/07/31 16:40:34.599|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.594|INFO |DataGateway.Configuration|DBConfiguration.xml – skipLegacyElasticConfiguration? False
2021/07/31 16:40:34.631|SLDBConnection|SLDBConnection|DBG|0|1|Updating TTL file.
2021/07/31 16:40:34.695|SLDBConnection|SLDataGateway.Repositories|INF|0|20|2021-07-31T16:40:34.695|DEBUG|Repository.MetaRepositoryServer|Server.Main|—————————————-START—————————————–
2021/07/31 16:40:34.700|SLDBConnection|SLDataGateway.Repositories|INF|0|14|2021-07-31T16:40:34.700|DEBUG|Repository.MetaRepositoryServer|DataGatewayRequestResponseServer/ClientToServerMetaRepositoriesChannel (1).CreateCommunicationChannels|Start incoming comm channel.
2021/07/31 16:40:34.771|SLDBConnection|SLDataGateway.Repositories|INF|0|21|2021-07-31T16:40:34.771|DEBUG|Repository.MetaRepositoryServer|DataGatewayRequestResponseServer/ClientToServerMetaRepositoriesChannel (1)|Waiting for connection on input channel
2021/07/31 16:40:34.774|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.773|DEBUG|DataGateway.DBGateway|Creating SwitchoverBuilderPartialA
2021/07/31 16:40:34.784|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.784|TRACE|DataGateway.FileOffload|Creating SQLiteConnection
2021/07/31 16:40:34.932|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.932|TRACE|DataGateway.FileOffload|Creating jobs table
2021/07/31 16:40:34.932|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.932|TRACE|DataGateway.FileOffload|Creating index on jobs table [1/2]
2021/07/31 16:40:34.932|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.932|TRACE|DataGateway.FileOffload|Creating index on jobs table [2/2]
2021/07/31 16:40:34.933|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.932|TRACE|DataGateway.FileOffload|Removing failed jobs
2021/07/31 16:40:34.952|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.952|INFO |DataGateway.FileOffload.SQLiteStorage|Amount of jobs in file offload storage: 0 Size used: 0 MB
2021/07/31 16:40:34.959|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.959|DEBUG|DataGateway.DBGateway|No FileOffloadConfiguration defined, falling back to default config: FileOffloadConfiguration[MaxSizeMB=10000]
2021/07/31 16:40:34.965|SLDBConnection|DBGateway.Initialize|DBG|0|1|Setting up connections and storage…
2021/07/31 16:40:36.044|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:36.044|INFO |DataGateway.Cassandra.ConnectionComposing|Composing Cassandra single node connection.
2021/07/31 16:40:36.102|SLDBConnection|CassandraConnection::BuildCluster|INF|0|1|Cluster with contactpoints: localhost and consistencylevel: One
2021/07/31 16:40:36.222|SLDBConnection|SLDBConnection|INF|0|1|Replication factor: 1
2021/07/31 16:40:36.225|SLDBConnection|CassandraConnection::Connect|INF|0|1|Attempting to connect to Cassandra…
2021/07/31 16:40:37.782|SLDBConnection|SLDBConnection|INF|0|1|CassandraConnection – connected to cluster with ips: 127.0.0.1:9042
2021/07/31 16:40:37.784|SLDBConnection|CassandraConnection::Connect|INF|0|1|Connection successful
2021/07/31 16:40:38.629|SLDBConnection|SLDBConnection|INF|0|1|Cassandra is currently running on version: 3.7

Jose Araujo [DevOps Advocate] Selected answer as best 3rd August 2021

3 Answers

0
19.27K 3 Comments

Hi Jose,

I was wondering, did you check first if you are able to query the Elasticsearch node from the server where the DMA is running? From a web browser (running from the server), can you perform the following query (based on your configuration):

http://35.175.118.112:9200/

This query should return information about the Elastic node. In case you are not able to retrieve any information, can you check if there is a firewall rule for port 9200?

Edit: Including a sample of DB.xml

<DataBases xmlns=”http://www.skyline.be/config/db”>
<DataBase active=”true” local=”true” type=”Cassandra”>
<ConnectString/>
<DBServer>localhost</DBServer>
<DSN/>
<DB>SLDMADB</DB>
<UID>root</UID>
<PWD>myPassword</PWD>
</DataBase>
<DataBase active=”true” search=”true” type=”Elasticsearch”>
<DBServer>10.11.12.13</DBServer>
<UID></UID>
<PWD></PWD>
</DataBase>
</DataBases>

Jose Araujo [DevOps Advocate] Selected answer as best 3rd August 2021
Jose Araujo [DevOps Advocate] commented

Hi,

Yes, can access to 9200 port from my DMA, I think the issue is with db.xml config.
Miguel Obregon [SLC] [DevOps Catalyst] commented

Hi Jose,
If possible, please could you insert these images as part of your question? (you can edit your question)
1
50 0 Comments

Hi Jose,
the DB.XML looks fine at first sight. I don’t see any errors in the DBConnection logging you posted.
I would have a look at the SLSearch logging for any pointers.

Bert Vervaele [SLC] Answered question 2nd August 2021
0
912 2 Comments

Jose,

DB.xml needs to include the configuration for Cassandra and also Elasticsearch if you are planning to use the latest version of the DataMiner. If what you posted is the complete DB.xml then that’s incorrect.

Please review the documentation at

Expert Hub – Elasticsearch

https://help.dataminer.services/dataminer/#t=DataMinerUserGuide%2Fpart_7%2FSkylineDataminerFolder%2FDB_xml.htm&rhsearch=elasticsearch&rhhlterm=elasticsearch&rhsyns=%20

Srikanth Mandava [SLC] [DevOps Advocate] Posted new comment 3rd August 2021
Jose Araujo [DevOps Advocate] commented

Hi,

Thanks for quick reply and thanks for all your support, I have 2 question

1. Need some changes or add something in my external server with elastic search?

2. You have template with db.xml with include casandra and elasticsearch?

Thanks
Srikanth Mandava [SLC] [DevOps Advocate] commented

Miguel already responded to these questions, and I see that this is now solved. Thanks.