Hi everyone,
I am working on an Automation script to retrieve all data from a logger table. According to the documentation (Querying logger tables | DataMiner Docs), this should be achievable using the GetPartialTableMessage class. However, my script consistently retrieves only 1,000 rows, which is far fewer than the full dataset I expect.
To address this, I added the forceFullTable=true option in the filter field, as suggested in the documentation. Unfortunately, this does not seem to make any difference — I still receive the same subset of 1,000 rows, regardless of whether forceFullTable=true is included or not.
My questions are:
- Are there any additional steps or configurations I might be missing specifically for querying logger tables?
- Is there a limitation or best practice for handling large datasets with
GetPartialTableMessagethat I should be aware of? - Could this behavior be related to the logger table itself, such as specific restrictions on row retrieval?
Any guidance or tips would be greatly appreciated.
Thanks in advance!
Hi,
Retrieving the full table content is not possible because of the underlying architecture of the different technologies in the background.
For example with Cassandra, data is stored with the partition key. To be able to query the data, a partition key must always be specified so it knows which partition the data should be looked up. I know there are full scan techniques (like allow filtering in the query), but these are not advised.
A logger table should be considered as big data which should not be fully loaded, hence why they were added as a logger table instead of a regular table. If the logger table needs to be queried then the partition key value has to be defined in the GetPartialTableMessage. The subset result will be limited to 1000 rows as that is also the default size that is kept in memory of the displayed table.
This means that there is a limitation of the database that returns a limited amount of rows, and a limitation in the table definition as that is designed to only keep 1000 rows (by default) in memory.
Regards,
Retrieving a complete logger table is not possible, as noted by Laurens Moutton. A possible workaround is to apply filters to the query in the request to narrow down the results. An example implementation could be as follows:
GetPartialTableMessage request = new GetPartialTableMessage
{
DataMinerID = DmaId,
ElementID = ElementId,
ParameterID = 1000, // Table ID
Filters = new[]
{
string.Format("value={0} == {1}", Column1, stringToFind1),
string.Format("value={0} == {1}", Column2, stringToFind2),
},
};