I'm currently using a product with a custom Protocol which has updated some of their API endpoints. Although the new endpoints are decoding properly in Postman requests, they are now showing up as random characters in my connector logs even before running through the JSON parser. I've traced it to the fact that they are using Brotli instead of GZip to compress the response body and I am seeing what looks like raw compressed data. Is there an easy way to get the decompressed content?
Also to note, I can't even take the response body and decode it myself since it is not capturing the entire body. In an example request, the expected response is a large JSON object (10.95 KB according to Postman), but when calling GetParameter I am getting a string of length 24.
Here is the XML for this particular request:
.<HTTP>
. <Session id="104" name="Get Data" loginMethod="credentials" userName="100" password="101">
. <Connection id="1">
. <Request verb="GET" url="/data"/>
. <Response statusCode="304">
. <Content pid="404"/>
. </Response>
. </Connection>
. </Session>.</HTTP>
Parameter 404 contains: [ýEýq@ 2NGQ&iýTýaý 8ý
Thanks!
This was never resolved. We just skipped including information from this endpoint and are requesting that the vendor honors the accept-encoding header in a future release.
Do you still want us to look into this further, or is this no longer needed?
It's not a high priority, but it would be helpful if this could be looked into.
Hi Nick,
Could you check if instead of using a GetParameter call, calling the GetData method from within your QAction would return a byte array with the expected size?
The reasoning is that compressed data very often contains the byte with the value 0, which, when parsed to a string, indicates its termination.
Although I am not too familiar with the specifics of Brotli, you should be able to pass the binary data to the library and have it decoded.
Unfortunately, that did not work. GetData returns a Byte Array but it is still of length 24. What’s strange is Brotli decompression is normally handled by default in .NET. I’m going to check the versioning on my .NET frameworks and nugets since I know it is a relatively new feature.
Do you happen to know if the backend on Dataminer uses the standard HttpClient class to handle these requests? And if it sets the HttpClientHandler.AutomaticDecompression property to something besides the default (None)?
Do note that QActions in DataMiner are compiled with .NET Framework and not .NET so libraries may not be available by default.
DataMiner’s backend for polling if you use HTTP sessions defined in the XML (see https://docs.dataminer.services/develop/schemadoc/Protocol/Protocol.HTTP.html) is written in C++ so it does use slightly different logic than what is available in .NET
But if you can define any header like “gzip, deflate” similar to the example then DataMiner should also be able to process it.
Got it, I just ran a version check within the Protocol and I see the problem. It is using v4.0.0.0 of System.Net.Http, Brotli decompression was added as a default in 4.3.2 or can be added manually in 4.1.2 (I believe).
If I update the version of .NET Framework on the server should that fix this? Or are these libraries statically linked?
I see that this question has been inactive for some time. Do you still need help with this? If not, could you select the answer (using the ✓ icon) to indicate that the question is resolved?