When setting huge numbers received from a device (up to 17 digits) to a protocol parameter, DataMiner is not showing the value correctly.
As a test we’re setting the ulong.MaxValue to both a numeric-double and other-string parameter to show the difference in value.
Is there a maximum range on what a protocol parameter can handle as an actual number?
If so, what’s the range?
Hi,
In the background, a parameter is either a string or a double. If it’s a number then this is a double in the background (8 bytes). A double consists of 3 parts: the sign bit, the exponent (11 bits) and fraction (52 bits). The combination of this makes that a double can also represent decimals because it’s a calculation that happens with the different parts.
When trying to write an ulong.MaxValue, this means that a 64 bits unsigned integer (8 bytes) will be tried to be written into a double with the same number of bytes where the double is making the calculation with the exponent and fraction, which is not as accurate as the unsigned integer for large numbers.
The maximum value that an integer can store in a double without losing precision would be 2^53 (roughly a 9 followed by 15 zeros, so 17 digits will not be possible to display such a large integer with the correct precision). A nice explanation can be found here .
Regards,