Question

Solved1.20K views3rd August 2021

1

Jose Araujo [DevOps Member]44 31st July 2021 0 Comments

Hi,

I have an issue when try to connect to external elasticsearch 6.8 with DMA, I'm using this DBConfiguration.xml config.

<?xml version="1.0"?>
<DataBases xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://www.skyline.be/config/db">
<DataBase active="true" local="true" type="Elasticsearch">
<ConnectString/>
<DBServer>35.175.118.112</DBServer>
<DSN/>
<DB></DB>
<UID>elastic</UID>
<PWD>changeme</PWD>
</DataBase>
</DataBases>

I get this error when restart DMA.

2021/07/31 16:40:33.765|SLLog|ApplyLogLevels|CRU|-1|Log Levels: Info: 0; Error: 0; Debug: 0 (from LogSettings.xml)
2021/07/31 16:40:33.769|SLDBConnection|SLDBConnection|INF|0|1|New DBGateway COM object created
2021/07/31 16:40:33.840|SLDBConnection|SLDBConnection|INF|0|1|Setting up connection to SLNet (Initial connection)
2021/07/31 16:40:34.407|SLDBConnection|SLDBConnection|INF|0|1|SLNet Connection has been established (da68abf6-7424-44c9-ace3-9482cab67ef8)
2021/07/31 16:40:34.583|SLDBConnection|DBGateway.Initialize|DBG|0|1|Loading configuration data...
2021/07/31 16:40:34.599|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.594|INFO |DataGateway.Configuration|DBConfiguration.xml - skipLegacyElasticConfiguration? False
2021/07/31 16:40:34.631|SLDBConnection|SLDBConnection|DBG|0|1|Updating TTL file.
2021/07/31 16:40:34.695|SLDBConnection|SLDataGateway.Repositories|INF|0|20|2021-07-31T16:40:34.695|DEBUG|Repository.MetaRepositoryServer|Server.Main|----------------------------------------START-----------------------------------------
2021/07/31 16:40:34.700|SLDBConnection|SLDataGateway.Repositories|INF|0|14|2021-07-31T16:40:34.700|DEBUG|Repository.MetaRepositoryServer|DataGatewayRequestResponseServer/ClientToServerMetaRepositoriesChannel (1).CreateCommunicationChannels|Start incoming comm channel.
2021/07/31 16:40:34.771|SLDBConnection|SLDataGateway.Repositories|INF|0|21|2021-07-31T16:40:34.771|DEBUG|Repository.MetaRepositoryServer|DataGatewayRequestResponseServer/ClientToServerMetaRepositoriesChannel (1)|Waiting for connection on input channel
2021/07/31 16:40:34.774|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.773|DEBUG|DataGateway.DBGateway|Creating SwitchoverBuilderPartialA
2021/07/31 16:40:34.784|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.784|TRACE|DataGateway.FileOffload|Creating SQLiteConnection
2021/07/31 16:40:34.932|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.932|TRACE|DataGateway.FileOffload|Creating jobs table
2021/07/31 16:40:34.932|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.932|TRACE|DataGateway.FileOffload|Creating index on jobs table [1/2] 2021/07/31 16:40:34.932|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.932|TRACE|DataGateway.FileOffload|Creating index on jobs table [2/2] 2021/07/31 16:40:34.933|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.932|TRACE|DataGateway.FileOffload|Removing failed jobs
2021/07/31 16:40:34.952|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.952|INFO |DataGateway.FileOffload.SQLiteStorage|Amount of jobs in file offload storage: 0 Size used: 0 MB
2021/07/31 16:40:34.959|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:34.959|DEBUG|DataGateway.DBGateway|No FileOffloadConfiguration defined, falling back to default config: FileOffloadConfiguration[MaxSizeMB=10000] 2021/07/31 16:40:34.965|SLDBConnection|DBGateway.Initialize|DBG|0|1|Setting up connections and storage...
2021/07/31 16:40:36.044|SLDBConnection|SLDataGateway.Repositories|INF|0|1|2021-07-31T16:40:36.044|INFO |DataGateway.Cassandra.ConnectionComposing|Composing Cassandra single node connection.
2021/07/31 16:40:36.102|SLDBConnection|CassandraConnection::BuildCluster|INF|0|1|Cluster with contactpoints: localhost and consistencylevel: One
2021/07/31 16:40:36.222|SLDBConnection|SLDBConnection|INF|0|1|Replication factor: 1
2021/07/31 16:40:36.225|SLDBConnection|CassandraConnection::Connect|INF|0|1|Attempting to connect to Cassandra...
2021/07/31 16:40:37.782|SLDBConnection|SLDBConnection|INF|0|1|CassandraConnection - connected to cluster with ips: 127.0.0.1:9042
2021/07/31 16:40:37.784|SLDBConnection|CassandraConnection::Connect|INF|0|1|Connection successful
2021/07/31 16:40:38.629|SLDBConnection|SLDBConnection|INF|0|1|Cassandra is currently running on version: 3.7

Jose Araujo [DevOps Member] Selected answer as best 3rd August 2021

3 Answers

0

Miguel Obregon [SLC] [DevOps Catalyst]18.56K Posted 2nd August 2021 3 Comments

Hi Jose,

I was wondering, did you check first if you are able to query the Elasticsearch node from the server where the DMA is running? From a web browser (running from the server), can you perform the following query (based on your configuration):

http://35.175.118.112:9200/

This query should return information about the Elastic node. In case you are not able to retrieve any information, can you check if there is a firewall rule for port 9200?

Edit: Including a sample of DB.xml

<DataBases xmlns="http://www.skyline.be/config/db">
<DataBase active="true" local="true" type="Cassandra">
<ConnectString/>
<DBServer>localhost</DBServer>
<DSN/>
<DB>SLDMADB</DB>
<UID>root</UID>
<PWD>myPassword</PWD>
</DataBase>
<DataBase active="true" search="true" type="Elasticsearch">
<DBServer>10.11.12.13</DBServer>
<UID></UID>
<PWD></PWD>
</DataBase>
</DataBases>

Jose Araujo [DevOps Member] Selected answer as best 3rd August 2021

Jose Araujo [DevOps Member] commented 2nd August 2021

Hi,

Yes, can access to 9200 port from my DMA, I think the issue is with db.xml config.

Jose Araujo [DevOps Member] commented 2nd August 2021

I have some errors

https://ibb.co/PhBJXH6
https://ibb.co/5KCqG33
https://ibb.co/4NSP1hc

Miguel Obregon [SLC] [DevOps Catalyst] commented 2nd August 2021

Hi Jose,
If possible, please could you insert these images as part of your question? (you can edit your question)

Hi,

Yes, can access to 9200 port from my DMA, I think the issue is with db.xml config.
I have some errors

https://ibb.co/PhBJXH6
https://ibb.co/5KCqG33
https://ibb.co/4NSP1hc
Hi Jose,
If possible, please could you insert these images as part of your question? (you can edit your question)

score 1 · Answer 1 · 2021-08-02T07:37:50+00:00

Hi Jose,
the DB.XML looks fine at first sight. I don't see any errors in the DBConnection logging you posted.
I would have a look at the SLSearch logging for any pointers.

score 0 · Answer 2 · 2021-08-02T16:16:51+00:00

Jose,

DB.xml needs to include the configuration for Cassandra and also Elasticsearch if you are planning to use the latest version of the DataMiner. If what you posted is the complete DB.xml then that's incorrect.

Please review the documentation at

Expert Hub – Elasticsearch

https://help.dataminer.services/dataminer/#t=DataMinerUserGuide%2Fpart_7%2FSkylineDataminerFolder%2FDB_xml.htm&rhsearch=elasticsearch&rhhlterm=elasticsearch&rhsyns=%20

Hi,

Thanks for quick reply and thanks for all your support, I have 2 question

1. Need some changes or add something in my external server with elastic search?

2. You have template with db.xml with include casandra and elasticsearch?

Thanks
Miguel already responded to these questions, and I see that this is now solved. Thanks.

Connect external Elasticsearch with DMA

3 Answers