Hi,

We have a DMS that spans two sites, with a fast private network between them. Our Cassandra cluster is configured with two Datacenters, each DC has two servers. Replication is configured such that every node has a full copy of the data.

We have a total of 20 DMAs, 10 on each site, split into two availablity zones on each site, 5 nodes in each. The cassandra hosts are also split across each availability zone. in DB.xml consistencyLevel = One. The DB.xml also lists all Cassandra nodes, not just their local ones. Any DMA can talk to any Cassandra node.

We've been experimenting with fault tolerance.

1.) Shutting down one node on a site

We found we were unable to create elements on the site that was missing one out of two nodes Dataminer produced the unable to save element dialogue. Creating elements on the other site was successful.

As all three other cassandra nodes were up and accessible, we was expecting this to work, as only one node has to report data written successfully

Restarting a DMA on the site that was missing a Cassandra node caused the DMA to connect to the other site successfully, and we were then able to create elements on the DMA that was restarted.

This error was also seen:-

Unexpected Exception [Unable to successfully create a storage with ID 6433f867-aec3-4c9a-9b7d-077826bb596d:
An exception occurred when creating a storage: Unable to perform authorization of login permission: Cannot achieve consistency level LOCAL_QUORUM]:
Parameter Notification Thread (   at Skyline.DataMiner.Core.StorageModule.Storages.StorageFactory.<GetStorage>d__11`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Skyline.DataMiner.Core.DMAObjects.Config.Elements.Storages.ElementStorageFactory.CreateDescriptionStorage(Guid storage)
   at Skyline.DataMiner.Core.DMAObjects.Config.Elements.Storages.ElementCRUD.Initialize()
   at Skyline.DataMiner.Net.MessageBuilder.BuildElementInfoEventMessage(Object[] elementInfo)
   at Skyline.DataMiner.Net.MessageHandler.ElementInfoToMessage(Object[] elementInfo, Dictionary`2 tmpCache)
   at Skyline.DataMiner.Net.MessageHandler.BuildSLNetCOMNotifications(SLNetRequestType type, Int32 subType, Int32 subType2, Object params1, Object params2)
   at Skyline.DataMiner.Net.Notifications.Parameters.ParameterNotificationManager.ExecuteItem(NotificationQueueItem itemToExecute))

I'm confused as to why it's trying for LOCAL QUORUM, with consistencyLevel is set to One.
2.) Shutting down both Cassandras at one site, prevented Datamienr from creating elements anywhere

We was expecting the other site to still allow element creation on the hosts that could communicate with the Cassandras on the other site.

The DB.xml specifies the hosts as follows on Site A:-

<DBServer>A1,A2,B1,B2</DBServer>

On Site B

<DBServer>B1,B2,A1,A2</DBServer>

Our understanding was that with consistencyLevel = One, that so long as a configured node was contactable reads and writes should be successful, with data replicated later, once the other nodes had rejoined the cluster.

In Cassandra, we have applied the following setting to the keyspaces

{'class' : 'NetworkTopologyStrategy','A' : 2,'B' : 2}

What can we do to make this system more fault tolerant, allowing for the scenario where two cassandra nodes are offline?