I was initially able to successfully install Opensearch in a separate EC2 instance as prescribed. However I faced issue with setting up keystore and truststore p12, as the service would not restart properly. In doing some webresearch, I was asked to remove the keystore, truststore passwords from opensearch.yml and instead place them securely in opensearch.keystore. Despite doing this, it did not work. So I attempted a fresh re-install of opensearch but facing the following error. I suspect opensearch is broken since I dont see opensearch.yml and jvm.options files getting created as part of installation anymore. Here is the installation result. Can you pls take a look and help me troubleshoot this issue?
sudo apt-get remove --purge opensearch
sudo apt-get update
sudo apt-get install opensearch
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
opensearch*
0 upgraded, 0 newly installed, 1 to remove and 145 not upgraded.
1 not fully installed or removed.
After this operation, 1212 MB disk space will be freed.
Do you want to continue? [Y/n] Y
(Reading database ... 115297 files and directories currently installed.)
Removing opensearch (2.18.0) ...
Running OpenSearch Pre-Removal Script
Processing triggers for libc-bin (2.39-0ubuntu8.3) ...
(Reading database ... 113869 files and directories currently installed.)
Purging configuration files for opensearch (2.18.0) ...
dpkg: warning: while removing opensearch, directory '/var/log/opensearch' not empty so not removed
Scanning processes...
Scanning linux images...
Running kernel seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
Hit:1 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable InRelease
Hit:5 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
opensearch
0 upgraded, 1 newly installed, 0 to remove and 145 not upgraded.
Need to get 0 B/917 MB of archives.
After this operation, 1212 MB of additional disk space will be used.
Selecting previously unselected package opensearch.
(Reading database ... 113832 files and directories currently installed.)
Preparing to unpack .../opensearch_2.18.0_amd64.deb ...
Running OpenSearch Pre-Installation Script
Unpacking opensearch (2.18.0) ...
Setting up opensearch (2.18.0) ...
Running OpenSearch Post-Installation Script
ERROR: Something went wrong during demo configuration installation. Please see the logs in /var/log/opensearch/install_demo_configuration.log
dpkg: error processing package opensearch (--configure):
installed opensearch package post-installation script subprocess returned error exit status 1
Processing triggers for libc-bin (2.39-0ubuntu8.3) ...
Errors were encountered while processing:
opensearch
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)
Thank you. I did, and found the following message.
OpenSearch install type: rpm/deb on Linux 6.8.0-1019-aws amd64
OpenSearch config dir: /etc/opensearch/
OpenSearch config file: /etc/opensearch/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.18.0
Detected OpenSearch Security Version: 2.18.0.0
No custom admin password found. Please provide a password via the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD.
So I did sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD=<custom password> systemctl start opensearch
I still see the same error in demo_configuration.log and additionally see some errors related to no SSL configuration found in /var/log/opensearch/opensearch.log and
"/var/log/opensearch/opensearch_server.json" access control exception in journalctl logs. Is there a way to attach log files here?
For OpenSearch versions 2.12 and later, you have to define the admin pw in the install command. If no pw is defined the initial demo security configuration will fail.
Example: sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD=<custom-admin-password> apt-get install opensearch
Try uninstalling and reinstalling using the correct command. See Install OpenSearch from an APT respository.
Regarding TLS configuration issues, double check your TLS settings in opensearch.yml. OpenSearch 2.18 has a known issue when using PKCS (p12) files.
Here is an example of a working TLS configuration in OpenSearch 2.18:
Jeremiah,
Thanks for the feedback. I did a fresh install on EC2 with initial admin password as suggested. I was able to get opensearch configured initially and query it as well.
{
"name" : "ip-10-168-4-205",
"cluster_name" : "opensearch",
"cluster_uuid" : "Oe-hu_HoRG2foM8_w64mLQ",
"version" : {
"distribution" : "opensearch",
"number" : "2.18.0",
"build_type" : "deb",
"build_hash" : "99a9a81da366173b0c2b963b26ea92e15ef34547",
"build_date" : "2024-10-31T19:15:31.009695761Z",
"build_snapshot" : false,
"lucene_version" : "9.12.0",
"minimum_wire_compatibility_version" : "7.10.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}
However, when I replaced the pem with p12 certs. I get the exception that its unable to read the p12 file even though opensearch user has permission 640 on the p12 cert, its in the correct folder referenced in opensearch.yml
Caused by: org.opensearch.OpenSearchException: Unable to read the file /etc/opensearch/cert/opensearchnode1-node-keystore.p12
. Please make sure this files exists and is readable regarding to permissions
at org.opensearch.security.ssl.config.SslCertificatesLoader.resolvePath(SslCertificatesLoader.java:165)
The OpenSearch user should be the owner of all your cert files. You can set the ownership of the "/etc/opensearch/cert" directory recursively to do this.
sudo chown -R opensearch:opensearch /etc/opensearch/cert
Thanks, it already was. Nevertheless I executed the command above. I see the same exception when restarting opensearch.
cd /etc/opensearch/cert
root@ip-10-168-4-205:/etc/opensearch/cert# ls -ls
total 8
8 -rw-r—– 1 opensearch opensearch 7498 Dec 17 15:34 opensearchnode1-node-keystore.p12
root@ip-10-168-4-205:/etc/opensearch# ls -ls
total 66028
4 drwxr-x— 3 opensearch opensearch 4096 Dec 13 19:24 aws
65936 -rwxr-x— 1 opensearch opensearch 67517785 Dec 16 18:48 awscliv2.zip
4 drwxr-x— 2 opensearch opensearch 4096 Dec 17 15:34 cert
4 drwxr-x— 2 opensearch opensearch 4096 Dec 17 15:31 config
4 -rwxr-x— 1 opensearch opensearch 3095 Dec 16 16:54 jvm.options
4 drwxr-x— 2 opensearch opensearch 4096 Oct 14 2022 jvm.options.d
20 -rwxr-x— 1 opensearch opensearch 17919 Oct 14 2022 log4j2.properties
8 -rwxr-x— 1 opensearch opensearch 6985 Dec 17 15:23 opensearch-bkup.yml
4 drwxr-x— 2 opensearch opensearch 4096 Dec 16 15:41 opensearch-notifications
4 drwxr-x— 2 opensearch opensearch 4096 Dec 16 15:41 opensearch-notifications-core
4 drwxr-x— 2 opensearch opensearch 4096 Dec 16 15:41 opensearch-observability
4 drwxr-x— 2 opensearch opensearch 4096 Dec 16 15:41 opensearch-performance-analyzer
4 drwxr-x— 2 opensearch opensearch 4096 Dec 16 15:41 opensearch-reports-scheduler
4 drwxr-x— 2 opensearch opensearch 4096 Dec 17 18:43 opensearch-security
4 -rwxr-x— 1 opensearch opensearch 196 Dec 16 15:43 opensearch.keystore
12 -rwxr-x— 1 opensearch opensearch 8896 Dec 18 01:11 opensearch.yml
Hi, can you please have a look at the /var/log/opensearch/install_demo_configuration.log log file? According to your installation result, there should be more info in there about the issues encountered during the demo configuration.