What is the impact of creating complex queries in dashboards for DataMiner?
Will there be an extra load on NATS/SLNet/Elastic/...?
Are there numbers on what the limitations are for queries or will the queries just be performed slower in case of many dashboards/queries?
Hi Michiel,
Great question!
It depends where the data is actually coming from. As you know, GQI allows you to query different data sets. We recently also added the possibility to create your own data set so the source of that could be anything.
GQI will start requesting data from the data source upon executing a query. For most sources, this will result in a new SLNet request which will be forwarded to the appropriate process. So this could be SLNet, SLElement, SLDataGateway or somewhere else.
Now GQI is designed with performance in mind. Here are some of the optimizations that are currently in place:
- Lazy loading of data (aka query sessions)
The client will start requesting data page by page. GQI will in turn also start requesting data from the source. If the client is only requesting 100 rows, then GQI will fetch data from the source until 100 rows can be returned.All operators are also built to be as lazy as possible, they will not request more data than required.
- Forwarding of filters
Filter operations that are done in the query are forwarded to the source. For instance, filtering a parameter table on a column is actually done in SLElement (or Elastic in case of logger tables). - Forwarding of aggregations
Aggregation operations that are doine in the query are forwarded to the source. For instance, aggregations on logger tables are forwarded to Elastic.
Note that there is currently no built-in mechanism of caching, this means that executing a query X times will cause the load X times.
Numbers on the amount of concurrent queries are hard since it highly depends of the query itself. These can be very complex and might require a lot of data.
No, there’s a limit to the amount of concurrent executing queries per DMA, this is currently set to 20. Any additional queries that are executed are placed in a queue.
So if you have many end-users consulting the dashboard at the same time you might overload the system? Can the system crash when performing too many queries, or is there a protection mechanism to prevent this from happening?