Hi all,
I have a QAction that runs every hour to monitor the software status of multiple devices. I maintain a table that stores the device URLs along with credentials and their API status. This QAction calls {DeviceURL}/api/system for each device and updates the status back in the table.
The issue is that the number of devices is dynamic and can vary, and currently processing the full table takes around 15 minutes because each endpoint is being hit and updated one by one. The list can be quite large, and the device count is not predictable. I’m trying to improve the performance and handle the calls more efficiently in batches. I cannot rely on a multithreaded timer since the number of endpoints is unknown.
At the moment, I’m creating tasks for each device and attempting to process them in batches. I’m looking for a better way to manage a large and dynamic number of HTTP calls with proper batching (update the rows in batch without much wait time to process the whole list) or throttling.
Any suggestions on how to restructure this for better performance would be really helpful.
public static void Run(SLProtocolExt protocol)
{
try
{
var devicesTable = Element.GetRows<DevicesQActionRow>(protocol, protocol.devices.TableId);
var tasks = new List<Task>();
var totalDevices = devicesTable.Count();
for (int i = 0; i < totalDevices; i += _batchSize)
{
var batch = devicesTable.Skip(i).Take(_batchSize).ToList();
tasks.AddRange(batch.Select(device => GetDeviceStatusAsync(device, protocol)).ToList());
Task.WhenAll(tasks);
}
protocol.Log($"Processed {tasks.Count} devices");
}
catch (Exception ex)
{
protocol.Log($"QA{protocol.QActionID}|{protocol.GetTriggerParameter()}|Run|Exception thrown:{Environment.NewLine}{ex}", LogType.Error, LogLevel.NoLogging);
}
}
private static async Task GetDeviceStatusAsync(DevicesQActionRow device, SLProtocolExt protocol)
{
var url = device.Deviceurl?.ToString();
var username = device.Deviceusername?.ToString();
var password = device.Devicepassword?.ToString();
var row = (DevicesQActionRow)protocol.devices.GetRow(device.Deviceid.ToString());
if (string.IsNullOrWhiteSpace(url))
{
return;
}
var path = new Uri(url + "/api/system");
try
{
var response = ExtensionMethods.GetResponseAsync(path, true, protocol, username, password).Result;
if (!string.IsNullOrWhiteSpace(response) && response != "[]")
{
JObject jsonObject = JObject.Parse(response);
var statusCode = jsonObject["statusCode"].ToString();
if (string.Equals(jsonObject["success"].ToString().ToLower(), "true", StringComparison.InvariantCultureIgnoreCase))
{
row.Deviceurl = device.Deviceurl;
row.Deviceapistatus = statusCode + " OK";
}
else
{
row.Deviceurl = device.Deviceurl;
row.Deviceapistatus = statusCode;
}
protocol.devices.SetRow(row);
}
}
catch (Exception ex)
{
row.Deviceurl = device.Deviceurl;
row.Deviceapistatus = "Error";
protocol.devices.SetRow(row);
}
}
Hi Srimathi,
I understand your concern about using a multithreaded timer when the number of endpoints is unknown. However, this is actually a good scenario to use one. A multithreaded timer ensures that each row in the table is processed at the defined interval, regardless of how many rows the table contains. Multithreaded timers are designed to handle such dynamic scenarios efficiently.
See the following page in the docs for more information:
https://docs.dataminer.services/develop/devguide/Connector/AdvancedMultithreadedTimers.html
https://docs.dataminer.services/develop/devguide/Connector/AdvancedMultithreadedTimersHttp.html.
Looking at the code snippet you shared, there are a couple of improvements that can be made:
1) Instead of calling the Result property after GetResponseAsync(), which makes it a synchrounous blocking call, it should be awaited:
var response = await ExtensionMethods.GetResponseAsync(path, true, protocol, username, password);
2) Task.WhenAll() returns a new task that completes when all provided tasks are finished, but in the current implementation, that task is not awaited. You can either use Task.WaitAll(tasks); or explicitly wait for completion like this:
Task.WhenAll(tasks).GetAwaiter().GetResult();