Hi Dojo - evaluating evolutions from the legacy SLNet towards gRPC and I'd like to check if the following could be a viable transition to get started:
- Cube → DMA = gRPC (for any server in the cluster)
- DMA ↔ DMA = existing SLNet communication
From this page, I understand that
so I'm wondering if gRPC is used for auto-forwarding events from the server to the client or if the client is still expected to request updates (e.g. as in SLNet polling mode).
In a big cluster environment where storm modes can be expected, I'd like to avoid the overhead of "polling" that slows down the client considerably - .NET Remoting with eventing would be the preferred mode pre-gRPC migrations, but any steer will be helpful.
Thanks
Hi Alberto,
Here is a comparsion of .NET Remoting and gRPC:
- .NET Remoting with eventing can deliver subscription messages instantly, but requires setting up a TCP connection from DMA to client, which is often not possible due to firewalls or NAT.
- .NET Remoting with polling uses "short polling", where the waiting happens on the clientside and the server answers immediately (either with some messages that were waiting to be delivered, or with an empty response). It is basically a tradeoff on how frequently the client will spam the server and how much extra delay you can tolerate for messages to be delivered. By default we have a slow polling cycle of 1000ms and a fast polling cycle of 100ms if the previous cycle received messages.
- In gRPC streaming the client performs a single HTTP request with an infinite response: the server uses "Transfer-Encoding: chunking" and delivers messages as soon as they occur, each in another chunk. However, this does not always work if there is a (reverse) proxy, gateway or deep-packet-inspecting firewall inbetween. These may either transform the random chunk sizes into fixed-length chunks, or buffer the chunking to try and convert it into a Content-Length response, or it may abort requests that take longer than x seconds. We have a detection mechanism for this, and if chunking is not working as expected, we fall back to polling (see next).
- gRPC with polling uses "long polling", where the waiting happens on the serverside. The client has an infinite loop of HTTP requests, and the server will wait up to 20 seconds until it has messages ready to be delivered. The advantage here is that there is no extra delay introduced as with "short polling", and that it does not rely on special HTTP features such as chunking or infinite responses. All subscription requests are typically pipelined on the same TCP connection, although that is not guaranteed (depends on the .NET HTTP connection pooling logic). Long polling is also used for Remote Cube access through *.on.dataminer.services.
Important to note is that we are using gRPC-web over HTTP/1.1 as opposed to "native" gRPC over HTTP/2 (where all traffic can be multiplexed over a single TCP connection) because we cannot yet guarantee support on all systems. (see https://learn.microsoft.com/en-us/aspnet/core/grpc/supported-platforms?view=aspnetcore-10.0 - mainly support for Windows Sever 2016/2019 and .NET Framework 4.x for clients such as Cube prevent the use of HTTP/2).
Let me know if you have any further questions.
The difference between gRPC streaming vs polling should be relatively low: the overhead is an extra HTTP request roundtrip inbetween sets of events, which will mostly be noticable on high-latency connections.
To be clear on the gRPC-web vs native gRPC: we currently only have gRPC-web implemented, so that applies even to setups that would be fully HTTP2 compatible.
Communication between agents is easiest (and recommended) to configure clusterwide with the EnableDotNetRemoting setting (https://docs.dataminer.services/dataminer/Administrator_guide/DataMiner_Agents/Configuring_a_DMA/Configuration_of_DataMiner_processes.html#disabling-net-remoting), but it is technically possible to configure the communication type between each pair of agents in either direction as desired, by creating <Redirect via="…" /> tags in DMS.xml on each agent (https://docs.dataminer.services/dataminer/Reference/Skyline_DataMiner_Folder/More_information_on_certain_files_and_folders/DMS_xml.html#redirects-subtag).
Thanks for the additional details on this, Bert – much appreciated
One last item please: is there any check we can do from CUBE (or from the DMA) to identify which rRPC mode is actually being used?
E.g. Within .NET Remoting & Cube default settings, we would notice the time spent before the client switched to polling – any recommended check to ensure gRPC with Streaming is used or at least preferred where available? Or is streaming something that can effective only when HTTP2 support will be included?
The internal state of the gRPC connection is not exposed. Aside from setting up deep packet inspection that decrypts HTTPS traffic so you can see the raw HTTP traffic, the simplest way to observe it from the outside is with a tool like Sysinternals TCPView or Nirsoft CurrPorts: when streaming is active, Cube will have an outgoing TCP connection to port 443 that only receives data, the sent packets/bytes will not update anymore.
There is no setting for it, it will be used unless the initial check for Transfer-Encoding: chunked check fails, so it is the preferred method. That check itself can take a few seconds, so while waiting for a positive outcome, polling is already used in order not have to wait on the first messages (whereas .NET Remoting had a 30 second delay before falling back to polling). If there is a direct end-to-end encrypted connection from client to DMA (=nothing to manipulate HTTP behavior) you can assume streaming will be used.
gRPC-web streaming is used over HTTP/1.1 with the limitation that it is unidirectional (from server to client). This is exactly what we need. Native gRPC over HTTP2 would also support client-to-server streaming and bidirectional streaming but we have no need for that.
https://learn.microsoft.com/en-us/aspnet/core/grpc/grpcweb?view=aspnetcore-10.0#grpc-web-and-streaming
Much appreciated – Brecht
I've logged this suggestion to possibly expose the gRPC mode in CUBE future versions:
https://community.dataminer.services/new-feature-suggestions/grpc-expose-in-cube-which-grpc-mode-is-in-use-streaming-vs-longpolling/
But for the time being I think we have sufficient info to define the migration steps with our squad and proceed with the staging environment.
Thanks also for all the testing suggestions you've shared – these will be helpful!
Marking as solved
Hi Alberto,
One thing that might already help clarify part of the picture: in recent Cube versions there’s an automatic switch to gRPC when you connect to a cloud setup.
So if the host is a cloud relay (anything ending in .dataminer.services), Cube will force gRPC, even if you explicitly selected something like Remoting. The reason is simply that cloud-connected agents don’t support .NET Remoting at all, so this guarantees the connection actually works.
For the other part of your question (eventing vs polling): this change is really just about how the connection is established. It doesn’t provide details yet on whether gRPC is already behaving like push/eventing or still follows a request-driven model internally.
Hope this helps a bit to position your migration approach
Thanks for this first feedback Mathias – I had done some searches in the meantime and found this older answer, where gRPC is recommended for new environments as it also "fixes polling / eventing issues as it work differently":
https://community.dataminer.services/question/improving-connection-to-cube/answer/122209/
Whether alarm updates in gRPC are carried through a single bidirectional stream, multiple streams, NATS-backed subscriptions, etc. – is the info I'm after – leaving the question open for the time being
Thanks for this breakdown Brecht – exactly the type of info I was after.
If I understand correctly where .NET Remoting with Eventing was possible before, moving to gRPC without streaming could limit the benefits of gRPC – so we'll keep this in mind for further optimizations.
Going to review the documentation you've shared on aspnet 10 – if this is linked ot the usage of Win 2019, likely we need to account for HTTP1.1
As for the "back-end" DMA to DMA communication, instead, is the switch to gRPC meant to happen at cluster level?