Hello,
I am setting up jmx remote access for our separate Cassandra node (4.0 running on Debian 11) and noticed that following this guide, the backup tool throws "not found" errors at me when I launch the command "StandaloneCassandraBackup.exe listsnapshot -h x.x.x.x" on our DMA, because of these added lines in the file cassandra-env.sh:
-Dcassandra.jmx.remote.port=7199 -Dcom.sun.management.jmxremote.rmi.port=7199 -Djava.rmi.server.hostname=x.x.x.x
So I removed them again and instead, followed this guide to set up jmx security and now the tool works. Due to this change and every time the backup or nodetool is used, credentials need to be provided (which still works and is OK for me).
When trying to take a snapshot however, the tool processes the series of commands without taking into account the credentials I add as arguments:
(Also when specifying the "–path C:\Program Files\nodetool.bat" everything following the space in "Program Files" gets omitted, so that does not work. I found out that the path is not even required as the tool automatically finds nodetool.bat, so I simply do not add the path argument)
Finally, to get the creation of snapshots via standalone tool to work I had to set the following in cassandra-env.sh (default is true):
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
Since in the guide "Making Cassandra nodetool accessible remotely" it nowhere says to disable jmx security, this simply cannot work the way it is described as jmx needs authentication by default.
In the end, I only had to disable jmx authentication and remove the lines which caused errors at the beginning, which does not correlate with the guide.
EDIT: To summarize, I either have to enable jmx authentication and always use credentials when issuing the standalone tool and nodetool commands, but backup will NOT work as it leaves out the credentials, or I have to disable jmx authentication and everything works, but this is not described in the Skyline guide.
Maybe some files changed over time and this has not been considered? Is the tool designed to work only without jmx authentication? Or have I done an error somewhere?
Regards,
Max
Howdy,
Your observations are majorly correct. I was able to reproduce this issue. (DMA 10.2.8 & Cassandra 4.0 node on Ubuntu)
- The tool is not working correctly when JMX authentication is enabled specifically for backup, clearsnapshot & refresh commands.
This can be seen below where the initial Status check ignores the credentials then the actual command does not. This causes a failure at the first point (status check) when JMX authentication is enabled. Please create a task in collaboration for this to be investigated further and addressed.
- By default the tool uses this path (C:\Program Files\Cassandra\bin\nodetool.bat) for Cassandra nodetool so you dont need to specify it if thats where its installed on your system. If however it is located elsewhere you need to specify as shown and it should work ok.
StandaloneCassandraBackup.exe listsnapshot -h 10.10.10.10 --path "D:\Program\Cassandra\bin\nodetool.bat"
- The documentation can indeed be improved to indicate clearly:
a) The actual full lines to be entered when updating the cassandra-env.sh file.JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.remote.port=7199"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=7199"
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=10.10.10.10"b) When making nodetool accessbile remotely, JMX authentication needs to be factored in. Addtional options (as you've shown) can be set in the cassandra-env.sh file to either enable or disable this.
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false" <Disables Authentiation which is enabled by default>
How to setup JMX authentication might be beyond the scope of the documentaion but maybe some external links can be provided.
(You can also help improve the documentaion if you wish). - I believe the tool is designed to work with or without Authentication. (If Point 1 is addressed).
- As you indicated everthing works if you disable authentication.
Hi Max
I think the intention is (can’t say for certain) to make it easy for the user to modify the environment file without necessarily having to understand or sift through all the code in it. By simply doing as outlined and declaring all those options again at the bottom of the file, you overwrite any preceding logic.
Without the RW permission in java set I am still able to connect so indeed maybe something peculiar in your setup.
Hi Wale, thanks for your input!
I agree to open a task about the first point.
As for point 3, these lines are even redundant because they already exist in the following if statement:
if [ “$LOCAL_JMX” = “yes” ]; then
JVM_OPTS=”$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT”
JVM_OPTS=”$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false”
else
JVM_OPTS=”$JVM_OPTS -Dcassandra.jmx.remote.port=$JMX_PORT”
JVM_OPTS=”$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT”
JVM_OPTS=”$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true”
The else statement is always true due to the “if [ “x$LOCAL_JMX” = “x” ]….” being commented out as described in the guide.
By the way I was pretty short-sighted about this, should have known that these lines are part of JVM_OPTS, alas I am no expert 🙂
What I also find strange is that jmx security has to be enabled (jmx.password file needs to be there and RW permissions for cassandra in java have to be set) because without it I get “connection refused” messages from the standalone tool, regardless of authentication being enabled or not. But that may be another issue on my part.