In our Cassandra cluster there is one – the same cert for each node – so for all Cassandra nodes it is one cert with CN: dma-cassandra.comp.local
And I got NoHostAvailableException when I enable TLS communication <TLSEnabled>true</TLSEnabled>
2023/02/01 13:18:57.410|SLDBConnection|CassandraConnection::Connect|ERR|0|1|Cassandra.NoHostAvailableException: All hosts tried for query failed (tried 10.44.222.171:9042: AuthenticationException ‘The remote certificate is invalid according to the validation procedure.’; 10.44.222.172:9042: AuthenticationException ‘The remote certificate is invalid according to the validation procedure.’; …), see Errors property for more info
Can I use one cert with one CN for all DB nodes or each Cassandra node must have separate cert and its CN must always match the machine’s hostname?
Could you please give me a hint regarding this?
====07.02.2023===== I added Wireshark screenshot
Hey Piotr,
These errors might occur when the IP/hostname of the Cassandra node is not included as a Subject Alternative Name (SAN) in your cert. The SAN field is what is used for verification and using Common Name (CN) to verify certs is deprecated (see RFC 6125).
The typical DB.xml configuration uses IPs, so DataMiner will try and match the IP to a SAN in the cert. If the IP isn’t present verification will fail. You can check your certificate SAN by simply opening the .crt/.cer file and going to the details tab. Or using keytool -printcert and checking the command output for SubjectAlternativeName.
If you find it isn’t present you will have to recreate your certificates and be sure to include -ext “san=ip:<NODE IP>” when creating the keystore and CSR. After this you would continue with the process of signing -> importing your certificates into the keystore. See below example.
Node hostname: node1.skyline.com
Node IP: 10.123.227.35
- Generate keystore
keytool -genkeypair -keyalg RSA -alias node1.skyline.com -keystore node1.jks -storepass PW -keypass PW -validity 365 -keysize 2048 -dname “CN=node1.skyline.com, OU=DMS, O=Skyline, C=US” -ext “san=ip:10.123.227.35” - Generate CSR
keytool -certreq -keystore node1.jks -alias node1.skyline.com -file node1.csr -keypass PW -storepass PW -ext “san=ip:10.123.227.35”
Regarding using one cert for all nodes, this isn’t recommended. Using one cert for all nodes is less secure than uniquely identifying each node using its own cert.
Hope this helps!
Regards,
Hey Piotr,
What does the Cassandra logging (system.log) show regarding connections? Maybe there is a problem with cipher suites between the servers.
Hi Jeremiah,
after I set in SAN for each node:
DNSName
IPAddress
it is working fine
thx
BTW
even if I set in
cassandra-env.sh
JVM_OPTS=”$JVM_OPTS -Djavax.net.debug=ssl”
I do not see any handshake with teh DMA client in system.log
Hi,
I would recommend doing a WireShark capture between the DMA and the Cassandra nodes. From there you will see the certificate that the Cassandra node(s) are sending to DataMiner. Depending on what is configured in your certificate (e.g. alternative names etc.) your certificate will be valid or not. What is used by DataMiner (hostname or IP) should match what is in the certificate.
[EDIT]
Example:
Hi Michiel,
thx
at the end of the handshake I got reset
Client -> Server [Client Hello]
Client <- Server [Server Hello]
Client Server [ClientKeyExchange][ChangeCipherSpec]
Client <- Server [ChangeCipherSpec][Encrypted Handshake Message]
Client <- Server [RST, ACK]
I also tried to enable debug info on Cassandra
cassandra-env.sh
JVM_OPTS=”$JVM_OPTS -Djavax.net.debug=ssl”
Hi Piotr,
I uploaded an example capture where you can see the certificates from the capture. This might help you to identify if the right certificates are used and what is in there.
At the beginning Jeremiah and Michiel thank you very much, your answer helped me to diagnose this case. Bellow short summary
If we have many cassandra nodes and we want to have one certificate for all nodes, we need add two values in Subject Alternative Name (SAN) for each node:
DNSName
IPAddress
Hi Jeremiah,
We have many Cassandra and elastic nodes, we prefer to generate one certificate for all nodes. I did as you said – I added ip of each node to SAN (I put Wireshark screenshot in my post) and I still have problem with dma connection to Cassandra cluster.
SLDBConnection|CassandraConnection::Connect|ERR|0|1|Cassandra.NoHostAvailableException: All hosts tried for query failed (tried 10.x.x.x:9042: AuthenticationException ‘The remote certificate is invalid according to the validation procedure.’;
I will add that using the same certificates, it connects correctly by Datastax dev Center client.