Hi Philip,

I believe a third site may be the solution here. On the condition that the sites have a low latency connection (<50ms) to each other.
The goal here is to have both sites be able to function independently of each other, is that correct?

With the above goal assumed. let's go over the various scenarios described here:

• What happens when the 2 main sites can't talk to each other and there's a tie-breaker?
  In this scenario, both main sites will have nodes that start an election process. 2 elections will take place which the tie-breaker node takes part in.
  The first election that finishes will result in a new master node being elected in one of the two main sites. This will invalidate the result of the slower 2nd election.
  Thus the tie-breaker + one of the main sites will form the reduced active cluster whilst the nodes in the other main site will remain inactive until the connection between the two sites has been restored.
  In this scenario, the DMS in the site that has gone dormant will no longer be able to use OpenSearch as they can only reach the tie-breaker which does not handle data, this halts their functionality and does not meet the goal above.
• What if we keep both sides online by reducing the settings?
  Here you are indeed introducing a split brain scenario. This can result in data loss or even a broken cluster  when the two sites are reunited once more.
  In this scenario, both sites can keep operating, but operations can be impacted as soon the connection is restored between the two sites. We recommend avoiding this situation at all costs.
• Is there an option to have the cluster go read-only?
  This would technically be feasible but is not supported and would require a different approach to how OpenSearch is utilized.
  
• Would we be better off splitting the clusters & writing to both?
  This is a valid option and would meet the goal stated above, when the connection to one of the two clusters would be interrupted the DMAs will indeed store the data in an offload file which will be offloaded to the OpenSearch as soon as the connection is restored. Do note however, DataMiner has a 10gb size limit to this file and does not have a mechanism to guarantee syncing of the data at runtime. If a longer outage would occur, the clusters would need to be brought in sync by restoring snapshots of the cluster that did not experience downtime to the other one. This requires downtime of both DMSs during this procedure.

My proposal would be to introduce a 3rd site, but ensure that it hosts one or more fully fledged OpenSearch nodes, not just a tie-breaker.
With this scenario we have a repeat of the one from the first bullet point, however due to the 3rd site having OpenSearch nodes that can handle data & ingest the DMS in the site that has gone dormant will continue to operate as it can reach OpenSearch nodes in the 3rd site with the capability of handling data & ingest. The main cost of this scenario is the need for multiple fully capable OpenSearch nodes on the third site.

If you've any further questions on the topic, feel free to ask here or reach out to me directly as we can set up a call to discuss the possible architectural options here.