Hello,
I was wondering how exactly SLDatagateway transfers the data taken from Dataminer and transfers that into a Cassandra Table.
I was asked if "it uses the same basic mechanism where all dbase updates are first put into a file, or multiple files, which are then process by a different process that is responsible for actually interfacing to the database? We need to understand how cassandra updates are handled by DM"
Thank you in advance for the detailed info!
Hello Ryan,
The "are first put into file" is probably a reference to mysql. But no, this isn't the case for Cassandra.
Data coming into Datagateway first gets stored in a queue and gradually this queue is being pushed to Cassandra trough queries. (this is all done in memory and goes pretty fast).
There's not much special happening in the flow from Datagateway to Cassandra. We use the Cassandra driver to compose the queries.
I hope this answers your question.
Hello,
That’s an intresting point. I can say that in older version (Pre 10.1.0, I can find the exact version if you are really intrested), this would indeed cause for some trouble, because deletes would not enter the queue. However, on the latest main release and feature releases, deletes are queued as well. It’s even so that in this case, the delete will remove the item from the queue, since it would be useless to write-delete it.
Thomas, that is great information and could explain why were are seeing orphan saved table entries in the database even after they are deleted from the actual table. Do you know if this same behavior also happens in MySQL? We are have this same problem with MySQL an Cassandra.
Thanks
Are all the dbase operations guaranteed to be processed in order? In other words if there are a bunch of operations processed on a saved parameter (Table) in DM will they get processed in sequential order? For example if a row is added to a saved table parameter and then deleted almost immediately with is there a guarantee that all the parameters for the table entry will fist get added to the dbase and then deleted or is it possible that the deletes happen before the data is stored in the database resulting in the table entry in question never gets deleted from the dbase.
Thanks