What is the advised method to use to avoid a massive load / memory usage when logic needs to retrieve DomInstances where the result could vary from 1 to 1 million (depending on the filter) ?
Option 1 (basic):
This will simply read All the DomInstances that matches the filter. So I'm expecting a big impact (timeout?) in case we are reading a lot of data.
public static List<DomInstance> GetAllDomInstanceByFieldValue(this DomHelper domHelper, FieldDescriptor fieldDescriptor, string value)
{
var filter = DomInstanceExposers.FieldValues.Field(fieldDescriptor.ID.Id.ToString()).Equal(value);
return domHelper.DomInstances.Read(filter);
}
Option 2 (paging):
This will get a set of DomInstances per go. (spreading the load somewhat)
public static List<DomInstance> GetAllDomInstanceByFieldValueByPages(this DomHelper domHelper, FieldDescriptor fieldDescriptor, string value)
{
var filter = DomInstanceExposers.FieldValues.Field(fieldDescriptor.ID.Id.ToString()).Equal(value);
var allDomInstances = new List<DomInstance>();
var pagingHelper = domHelper.DomInstances.PreparePaging(filter);
while (pagingHelper.MoveToNextPage())
{
var currentPage = pagingHelper.GetCurrentPage();
allDomInstances.AddRange(currentPage);
}
return allDomInstances;
}
How many items are retrieved via 1 page?
Option 3 (Skyline.DataMiner.Net.Tools):
I found this, but I don’t know what it does or when this should be used:
public static List<DomInstance> GetDomInstances(this DomHelper domHelper, List<Guid> instanceIds)
{
var instances = Tools.RetrieveBigOrFilter(
instanceIds,
id => DomInstanceExposers.Id.Equal(id),
filter => domHelper.DomInstances.Read(filter));
return instances;
}
Hi Mieke,
If retrieving a huge amount of DomInstances would be required, paging is the way to go. Keep in mind that the example you provided will load all these DomInstances into memory. If these could potentially be a million, you could push SLAutomation out of memory. In that case, it is recommended to get a page, do your actions, and then get the next page. That way you are only using the amount of memory that one 'chunk' needs.
Do note that this is not a definitive solution as it always depends on the use case, but the general rules to keep in mind are:
- Try to limit the size of the messages sent between server & client
- Try to keep the number of objects loaded into memory in mind