Script failure ‘ReservationUpdateException’

240 views
7
1 Comment

Hi there,

We had an issue were after attempting to run a script we saw the following failure message: Skyline.Dataminer.Library.Exceptions.ReservationUpdateException: Reservation named XXXXX wasn't updated successfully.

I can see this error in the SLAutomation log but I would like to continue to trace this through and would like some advice on what logs I could potentially check next to see why this reservation change for a booking failed to update?

Jorge Dias [SLC] [DevOps Enabler] Posted new comment
Jorge Dias [SLC] [DevOps Enabler] commented

Hi Philip, do you have the exception full stack trace? If so, can you post, should already contain some information of what happened

1 Answer

1
14.36K 2 Comments

Hi Philip,

You should be able to get more details on this in the resource manager logging.

Ive Herreman [SLC] [DevOps Enabler] Edited comment
Philip Greenhalgh [DevOps Advocate] commented

Hi there we had the failure again and we werent able to find any errors at the same time in resource manager but we were able to find errors in the booking manager.

At 22/01/2026 15:29:09.1526:

Resource Action failed due to:
(Code: 0x80131604) Skyline.DataMiner.Net.Exceptions.DataMinerException: Exception has been thrown by the target of an invocation. —> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. —> Skyline.DataMiner.Net.Exceptions.DataMinerException: Unexpected exception occurred —> Skyline.DataMiner.Net.Exceptions.DataMinerException: (Code: 0x80131500) Skyline.DataMiner.Net.Exceptions.DataMinerException: Unexpected exception occurred —> SLDataGateway.API.Types.Exceptions.Queries.QueryException: Exception thrown while executing paged query: Query(((ReservationInstance.End[DateTime] >=01/22/2026 15:28:26) AND (ReservationInstance.Start[DateTime] <=01/22/2026 15:28:26)) ) —> System.AggregateException: One or more errors occurred. —> Elasticsearch.Net.UnexpectedElasticsearchClientException: The operation has timed out. —> System.Net.WebException: The operation has timed out.
at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)
at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType)
at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType)
at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData)
at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
— End of inner exception stack trace —
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
at Elasticsearch.Net.ElasticLowLevelClient.Search[TResponse](String index, PostData body, SearchRequestParameters requestParameters)
at Nest.LowLevelDispatch.SearchDispatch[TResponse](IRequest`1 p, SerializableData`1 body)
at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[TRequest,TQueryString,TResponse](TRequest request, Func`3 responseGenerator, Func`3 dispatch)
at Skyline.DataMiner.SLSearch.Elastic6.StorageTypes.ElasticStorageType`1.HandleSearchRequest[TSearchType](SearchRequest request)
at Skyline.DataMiner.SLSearch.Elastic6.StorageTypes.ElasticCustomData`1.GetFirstPage(SearchRequest request)
at Skyline.DataMiner.SLSearch.Elastic6.StorageTypes.ElasticStorageType`1.GetFirstPage(QueryContainer query, IList`1 sort, Int32 pageSize, String searchTerm, String customAlias, Boolean isIdFilter, String idFilter)
at Skyline.DataMiner.SLSearch.Elastic6.PagingHandlers.ElasticPagingHandler`1.TryGetNextPageFromElastic(GetFirstPageFunc getFirstPageFunc, GetSubsequentPagesFunc getSubsequentPagesFunc, ParseDataFunc parseFunc, IEnumerable`1& page)
at Skyline.DataMiner.SLSearch.Elastic6.PagingHandlers.ElasticPagingHandler`1.<TryFillCacheAsync>d__59.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Skyline.DataMiner.SLSearch.Elastic6.PagingHandlers.ElasticPagingHandler`1.<>c__DisplayClass57_0.<<GetNextPageAsync>b__0>d.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Skyline.DataMiner.Net.SLDataGateway.Helpers.AsyncHelper.<TimeoutAfter>d__10`1.MoveNext()
— End of inner exception stack trace —
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Skyline.DataMiner.SLSearch.Elastic6.StorageTypes.ElasticStorageType`1.ReadPaged(ElasticPagingHandler`1 pagingHandler)
— End of inner exception stack trace —
at Skyline.DataMiner.SLSearch.Elastic6.StorageTypes.ElasticStorageType`1.ReadPaged(ElasticPagingHandler`1 pagingHandler)
at Skyline.DataMiner.SLSearch.Elastic6.StorageTypes.ElasticStorageType`1.ReadPagedDataInternal(APagingHandler`1 handler)
at SLDataGateway.Types.StorageType`1.ReadPagedData(PagingCookie cookie)
at SLDataGateway.Types.Storage.StorageTypeLimitPagingHandler`1.GetNextPage()
at SLCassandraClassLibrary.DBGateway.RepositoryUtil.Types.DataSetStorageTypeRepository`1.ReadPaged(PagingCookie cookie)
at SLCassandraClassLibrary.DBGateway.RepositoryMessageHandlerUtil.DataSetStorageTypeRepositoryMessageHandler`1.HandlePagedDataSetRequest(IDataSetStorageTypeRepository`1 repository, PagedDataSetRequest`1 pagedDataSetRequest, IDataConnectionInfo connectionInfo, ILogger logger)
at SLCassandraClassLibrary.DBGateway.RepositoryUtil.Factories.StorageTypeDatabaseRepositoryFactory.<>c__DisplayClass29_0`1.<GetDataSetRepositoryMessageHandlerBuilder>b__0(IDataSetRepository`1 repository, IDataSetRequest message, IDataConnectionInfo connInfo)
at SLDataGateway.API.Repositories.MessageHandlers.CloneUtil.RepositoryMessageHandler`1.HandleRequestInternal(TRepository repository, IDataSetRequest requestMessage, IDataConnectionInfo connectionInfo)
at SLDataGateway.API.Repositories.MessageHandlers.CloneUtil.RepositoryMessageHandler`1.HandleRequest(IBasicRepository basicRepository, IDataSetRequest requestMessage, IDataConnectionInfo connectionInfo)
at SLDataGateway.API.Repositories.ADatabaseRepository.HandleRequest(IDataSetRequest requestMessage, IDataConnectionInfo connectionInfo)
at SLDataGateway.API.Repositories.MessageHandler.ADataSetRequestMessageHandler.HandleRequest(IDataSetRequest dataSetRequest, IDataConnectionInfo connInfo)
— End of inner exception stack trace —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at SLDataGateway.API.Requests.Queries.Abstract.ADatabaseQuery`2.<InternalExecuteAsync>d__17.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at SLDataGateway.API.Requests.Queries.Abstract.ADatabaseQuery`2.<ExecuteAsync>d__12.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Skyline.DataMiner.Net.SLDataGateway.Helpers.AsyncHelper.WaitToComplete(Task task, TimeSpan timeout)
at SLDataGateway.API.Requests.Queries.Abstract.ADatabaseQuery`2.Execute()
at SLDataGateway.API.Paging.PagingHandlers.RepositoryPagingHandler`1.GetNextPage()
at SLDataGateway.API.Paging.PagingHandlers.RepositoryPagingHandler`1.NextPage()
at SLDataGateway.API.Repositories.MessageHandlers.DataSetUtil.DataSetRepositoryMessageHandler`1.GetNextPageInfo(IDataSetRepository`1 repository, PagedDataSetRequest`1 requestMessage, IDataConnectionInfo connectionInfo, IQuery`1 overruledQuery, ITracingConfiguration overruledTracingConfiguration)
at SLDataGateway.API.Repositories.MessageHandlers.DataSetUtil.DataSetRepositoryMessageHandler`1.HandlePagedDataSetRequest(IDataSetRepository`1 repository, PagedDataSetRequest`1 requestMessage, IDataConnectionInfo connectionInfo, IQuery`1 overruledQuery)
at Skyline.DataMiner.Net.SLDataGateway.API.Repositories.MessageHandler.LocalDatabaseRepositoryMessageHandler.Helper`1.HandlePagedDataSetRequest(IDataSetRepository`1 repository, PagedDataSetRequest`1 requestMessage, ILogger logger, IDataConnectionInfo connectionInfo, IQuery`1 overruledQuery)
at Skyline.DataMiner.Net.SLDataGateway.API.Repositories.MessageHandler.LocalDatabaseRepositoryMessageHandler.Helper`1.HandlePagedDataSetRequest(IDataSetRepository repository, PagedDataSetRequest requestMessage, ILogger logger, IDataConnectionInfo connectionInfo, IQuery overruledQuery)
at Skyline.DataMiner.Net.SLDataGateway.API.Repositories.MessageHandler.LocalDatabaseRepositoryMessageHandler.HandlePagedDataSetRequest[T](IDataSetRepository`1 repository, PagedDataSetRequest`1 requestMessage, ILogger logger, IDataConnectionInfo connectionInfo, IQuery`1 overruledQuery)
at Skyline.DataMiner.Net.SLDataGateway.API.Repositories.Factory.LocalDatabaseRepositoryFactory.<>c__DisplayClass8_0`1.<GetDataSetRepositoryMessageHandlerBuilder>b__2(IDataSetRepository`1 repository, IDataSetRequest message, IDataConnectionInfo info)
at SLDataGateway.API.Repositories.MessageHandlers.CloneUtil.RepositoryMessageHandler`1.HandleRequestInternal(TRepository repository, IDataSetRequest requestMessage, IDataConnectionInfo connectionInfo)
at SLDataGateway.API.Repositories.MessageHandlers.CloneUtil.RepositoryMessageHandler`1.HandleRequest(IBasicRepository basicRepository, IDataSetRequest requestMessage, IDataConnectionInfo connectionInfo)
at SLDataGateway.API.Repositories.ADatabaseRepository.HandleRequest(IDataSetRequest requestMessage, IDataConnectionInfo connectionInfo)
at Skyline.DataMiner.Net.SLDataGateway.API.Repositories.MessageHandler.DataSetRequestMessageHandler.HandleRequestInternal(IDataSetRequest dataSetRequest, IDataConnectionInfo connectionInfo)
at SLDataGateway.API.Repositories.MessageHandler.ADataSetRequestMessageHandler.HandleRequest(IDataSetRequest dataSetRequest, IDataConnectionInfo connInfo)
— End of inner exception stack trace —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at SLDataGateway.API.Requests.Queries.Abstract.ADatabaseQuery`2.<InternalExecuteAsync>d__17.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at SLDataGateway.API.Requests.Queries.Abstract.ADatabaseQuery`2.<ExecuteAsync>d__12.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Skyline.DataMiner.Net.SLDataGateway.Helpers.AsyncHelper.WaitToComplete(Task task, TimeSpan timeout)
at SLDataGateway.API.Requests.Queries.Abstract.ADatabaseQuery`2.Execute()
at Skyline.DataMiner.Net.Apps.Utils.CustomDataRepositoryStorage`2.InnerReadAll(IQuery`1 query)
at Skyline.DataMiner.Net.AsyncDatabaseCacher`2.ReadAll(IQuery`1 query)
at Skyline.DataMiner.Net.DatabaseCallerExtensions.ReadAll[T](IDatabaseCaller`1 db, FilterElement`1 filter)
at Skyline.DataMiner.Net.ResourceManager.Storage.ReservationInstanceTimeRangedCache.FillCacheWithTimeRange(TimeRangeUtc timeRange)
at Skyline.DataMiner.Net.ResourceManager.Storage.ReservationInstanceTimeRangedCache.GetMissingTimeRangesFromDatabase(List`1 requestedTimeRanges)
at Skyline.DataMiner.Net.ResourceManager.Storage.ReservationInstanceTimeRangedCache.GetByFilter(FilterElement`1 filter)
at Skyline.DataMiner.Net.ResourceManager.Storage.ReservationInstanceTimeRangedCache.TryResolveFilter(FilterElement`1 filterToResolve, IEnumerable`1& objects)
at Skyline.DataMiner.Net.ResourceManager.Storage.ReservationInstanceStorage.GetByFilter(FilterElement`1 filter)
at Skyline.DataMiner.Net.ResourceManager.ResourceManager.InnerGetReservationInstances(FilterElement`1 filter, Boolean& success, Boolean includeGeneratedUniques, Object context)
at Skyline.DataMiner.Net.Messages.ResourceManagerHelper.GetReservationInstances(FilterElement`1 filter)
at Skyline.DataMiner.Net.ResourceManager.ResourceManager.GetExcludedResources(Boolean& success, DateTime start, DateTime stop, Dictionary`2& resource2OverriddenProperties)
at Skyline.DataMiner.Net.ResourceManager.ResourceManager.InnerGetResources(Resource[] filters, Boolean& success, Object context, Boolean searchPools, Boolean searchProperties, Nullable`1 startTime, Nullable`1 stopTime)
at Skyline.DataMiner.Net.Messages.ResourceManagerHelper.InnerGetResources(GetResourceMessage msg, Object context)
— End of inner exception stack trace —
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Delegate.DynamicInvokeImpl(Object[] args)
at Skyline.DataMiner.Net.ResourceManager.ResourceManager.GetResourceManagerResponseMessage(IResourceManagerRequestMessage request, IConnectionInfo connInfo)
at Skyline.DataMiner.Net.ResourceManager.ResourceManager.ExecuteResourceManagerRequest(IResourceManagerRequestMessage request, IConnectionInfo connInfo)
at Skyline.DataMiner.Net.ResourceManager.ResourceManager.HandleResourceManagerRequest(IResourceManagerRequestMessage request, IConnectionInfo connInfo)
at Skyline.DataMiner.Net.Facade.HandleClientRequestMessage(IConnectionInfo connInfo, ClientRequestMessage oneMsg, Boolean canQueue)
at Skyline.DataMiner.Net.Facade.HandleMessageInternal(IConnectionInfo connInfo, DMSMessage oneMsg, Int32 groupID, Int32 groupTotal)
at Skyline.DataMiner.Net.Facade.HandleMessage(IConnectionInfo connInfo, DMSMessage oneMsg, Int32 groupID, Int32 groupTotal)
at Skyline.DataMiner.Net.BaseFacade.HandleMessages(IConnectionInfo connInfo, DMSMessage[] msgs)
at Skyline.DataMiner.Net.Facade.HandleImpersonateMessage(IConnectionInfo originalConnInfo, ImpersonateMessage im)
at Skyline.DataMiner.Net.Facade.HandleMessageInternal(IConnectionInfo connInfo, DMSMessage oneMsg, Int32 groupID, Int32 groupTotal)
at Skyline.DataMiner.Net.Facade.HandleMessage(IConnectionInfo connInfo, DMSMessage oneMsg, Int32 groupID, Int32 groupTotal)
at Skyline.DataMiner.Net.BaseFacade.HandleMessages(IConnectionInfo connInfo, DMSMessage[] msgs)
at Skyline.DataMiner.Net.Facade.HandleMessagesForClient(IConnectionInfo connInfo, DMSMessage[] messages)
at Skyline.DataMiner.Net.Transport.Remoting.InternalHandleMessages(ConnectionToken token, DMSMessage[] inputMessages, ConnectionTransportType ctt)
— End of inner exception stack trace —

Server stack trace:
at Skyline.DataMiner.Net.Transport.Remoting.InternalHandleMessages(ConnectionToken token, DMSMessage[] inputMessages, ConnectionTransportType ctt)
at Skyline.DataMiner.Net.Transport.Remoting.HandleMessages(ConnectionToken token, DMSMessage[] inputMessages, ConnectionTransportType ctt)
at Skyline.DataMiner.Net.Transport.Remoting.HandleMessages(ConnectionToken token, DMSMessage[] inputMessages)
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Skyline.DataMiner.Net.ISLNet.HandleMessages(ConnectionToken token, DMSMessage[] inputMessages)
at Skyline.DataMiner.Net.LegacyRemotingConnection.TransportSpecificHandleMessage(ConnectionToken tok, DMSMessage[] msgs, Int32 timeout)
at Skyline.DataMiner.Net.Connection.InternalHandlePackedMessages(ConnectionToken tok, DMSMessage[] msgs, Int32 timeout)
at Skyline.DataMiner.Net.Connection.InternalHandleUnpackedMessages(DMSMessage[] msgs, Int32 timeout)
at Skyline.DataMiner.Net.Connection.HandleMessages(DMSMessage[] msgs, Int32 timeout)
at Skyline.DataMiner.Net.Connection.HandleMessages(DMSMessage[] msgs)
at Skyline.DataMiner.Net.Impersonating.ImpersonateConnection.HandleSingleResponseMessage(DMSMessage msg)
at Skyline.DataMiner.Library.Solutions.SRM.SrmManagersWithLog.RequestResponseEvent(Object sender, IManagerRequestResponseEventArgs e)
at Skyline.DataMiner.Net.IManager.Helper.IManagerEventRaiser.RaiseEvent[T](EventHandler`1 handler, T args)
at Skyline.DataMiner.Net.Messages.ResourceManagerHelper.SendAndReceiveMessage(IManagerRequestResponseEventArgs& args)
at Skyline.DataMiner.Net.Messages.ResourceManagerHelper.GetResourcesFromServer(Resource[] filters, Boolean& success, Object context, Boolean searchPools, Boolean searchProperties, Nullable`1 startTime, Nullable`1 stopTime)
at Skyline.DataMiner.Net.Messages.ResourceManagerHelper.InnerGetResources(Resource[] filters, Boolean& success, Object context, Boolean searchPools, Boolean searchProperties, Nullable`1 startTime, Nullable`1 stopTime)
at Skyline.DataMiner.Net.Messages.ResourceManagerHelper.GetResources(Resource[] filters)
at Skyline.DataMiner.Library.Resource.ResourceManagerExtensionMethods.GetFunctionResources(ResourceManagerHelper resourceManager, IEnumerable`1 ids)
at SRM.ResourceActions.Script.AddAction.UpdateService(ServiceResourceUsageDefinition resourceUsageDefinition)
at SRM.ResourceActions.Script.AddAction.AddNodeToReservationResourceUsage(Node availableTargetNode)
at SRM.ResourceActions.Script.AddAction.PreProcessing(Engine engine)
at SRM.ResourceActions.Script.Run(Engine engine)

and at 22/01/2026 15:53:15.6764:

Resource Action failed due to:
Skyline.DataMiner.Library.Exceptions.ResourceManagerException: The service reservation instance 'Channel 4' could not be updated. The removal of the resource usage definition for node '9' failed. Exception: System.AggregateException: One or more errors occurred. —> Skyline.DataMiner.Net.Exceptions.DataMinerSecurityException: No such remote connection (7175a69884). Retried.

Server stack trace:
at Skyline.DataMiner.Net.Facade.HandleImpersonateMessage(IConnectionInfo originalConnInfo, ImpersonateMessage im)
at Skyline.DataMiner.Net.Facade.HandleMessageInternal(IConnectionInfo connInfo, DMSMessage oneMsg, Int32 groupID, Int32 groupTotal)
at Skyline.DataMiner.Net.Facade.HandleMessage(IConnectionInfo connInfo, DMSMessage oneMsg, Int32 groupID, Int32 groupTotal)
at Skyline.DataMiner.Net.BaseFacade.HandleMessages(IConnectionInfo connInfo, DMSMessage[] msgs)
at Skyline.DataMiner.Net.Facade.HandleMessagesForClient(IConnectionInfo connInfo, DMSMessage[] messages)
at Skyline.DataMiner.Net.Transport.Remoting.InternalHandleMessages(ConnectionToken token, DMSMessage[] inputMessages, ConnectionTransportType ctt)
at Skyline.DataMiner.Net.Transport.Remoting.HandleMessages(ConnectionToken token, DMSMessage[] inputMessages, ConnectionTransportType ctt)
at Skyline.DataMiner.Net.Transport.Remoting.HandleMessages(ConnectionToken token, DMSMessage[] inputMessages)
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Skyline.DataMiner.Net.ISLNet.HandleMessages(ConnectionToken token, DMSMessage[] inputMessages)
at Skyline.DataMiner.Net.LegacyRemotingConnection.TransportSpecificHandleMessage(ConnectionToken tok, DMSMessage[] msgs, Int32 timeout)
at Skyline.DataMiner.Net.Connection.InternalHandlePackedMessages(ConnectionToken tok, DMSMessage[] msgs, Int32 timeout)
at Skyline.DataMiner.Net.Connection.InternalHandleUnpackedMessages(DMSMessage[] msgs, Int32 timeout)
at Skyline.DataMiner.Net.Connection.HandleMessages(DMSMessage[] msgs, Int32 timeout)
at Skyline.DataMiner.Net.Connection.HandleMessages(DMSMessage[] msgs)
at Skyline.DataMiner.Net.Impersonating.ImpersonateConnection.HandleSingleResponseMessage(DMSMessage msg)
at Skyline.DataMiner.Library.Solutions.SRM.SrmManagersWithLog.RequestResponseEvent(Object sender, IManagerRequestResponseEventArgs e)
at Skyline.DataMiner.Net.IManager.Helper.IManagerEventRaiser.RaiseEvent[T](EventHandler`1 handler, T args)
at Skyline.DataMiner.Net.Messages.ResourceManagerHelper.SendAndReceiveMessage(IManagerRequestResponseEventArgs& args)
at Skyline.DataMiner.Net.Messages.ResourceManagerHelper.SetReservationInstancesOnServer(ReservationInstance[] reservationInstances, Boolean& success, Boolean isDelete, Object context, Boolean isSyncMessage, Boolean forceQuarantine, Boolean transactional, List`1 reservationsToIgnore, Boolean isGenerationForDefinition, Boolean validateSecurityViews)
at Skyline.DataMiner.Net.Messages.ResourceManagerHelper.InnerSetReservationInstances(ReservationInstance[] reservationInstances, Boolean& success, Boolean isDelete, Object context, Boolean isSyncMessage, Boolean forceQuarantine, Boolean transactional, Boolean validateSecurityViews)
at Skyline.DataMiner.Net.Messages.ResourceManagerHelper.<>c__DisplayClass119_0.<SetReservationInstances>b__0()
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
— End of inner exception stack trace —
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at Skyline.DataMiner.Net.Messages.ResourceManagerHelper.SetReservationInstances(ReservationInstance[] reservationInstances, Boolean& success, Boolean isDelete, Boolean forceQuarantine, Boolean transactional)
at Skyline.DataMiner.Library.Solutions.SRM.Helpers.ResourceManagerHelperExtended.<>c__DisplayClass9_0.<AddOrUpdateReservationInstances>b__0()
at Skyline.DataMiner.Library.Solutions.SRM.Helpers.ResourceManagerHelperExtended.RunQuery[T](Func`1 query, String queryInfo)
at Skyline.DataMiner.Library.Solutions.SRM.Helpers.ResourceManagerHelperExtended.AddOrUpdateReservationInstances(ReservationInstance[] reservationInstances)
at SRM.ResourceActions.Script.RemoveAction.RemoveResourceFromReservationInstance(ServiceReservationInstance reservationInstance, Node node)
—> (Inner Exception #0) (Code: 0x800402EB) Skyline.DataMiner.Net.Exceptions.DataMinerSecurityException: No such remote connection (7175a69884). Retried.

Server stack trace:
at Skyline.DataMiner.Net.Facade.HandleImpersonateMessage(IConnectionInfo originalConnInfo, ImpersonateMessage im)
at Skyline.DataMiner.Net.Facade.HandleMessageInternal(IConnectionInfo connInfo, DMSMessage oneMsg, Int32 groupID, Int32 groupTotal)
at Skyline.DataMiner.Net.Facade.HandleMessage(IConnectionInfo connInfo, DMSMessage oneMsg, Int32 groupID, Int32 groupTotal)
at Skyline.DataMiner.Net.BaseFacade.HandleMessages(IConnectionInfo connInfo, DMSMessage[] msgs)
at Skyline.DataMiner.Net.Facade.HandleMessagesForClient(IConnectionInfo connInfo, DMSMessage[] messages)
at Skyline.DataMiner.Net.Transport.Remoting.InternalHandleMessages(ConnectionToken token, DMSMessage[] inputMessages, ConnectionTransportType ctt)
at Skyline.DataMiner.Net.Transport.Remoting.HandleMessages(ConnectionToken token, DMSMessage[] inputMessages, ConnectionTransportType ctt)
at Skyline.DataMiner.Net.Transport.Remoting.HandleMessages(ConnectionToken token, DMSMessage[] inputMessages)
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Skyline.DataMiner.Net.ISLNet.HandleMessages(ConnectionToken token, DMSMessage[] inputMessages)
at Skyline.DataMiner.Net.LegacyRemotingConnection.TransportSpecificHandleMessage(ConnectionToken tok, DMSMessage[] msgs, Int32 timeout)
at Skyline.DataMiner.Net.Connection.InternalHandlePackedMessages(ConnectionToken tok, DMSMessage[] msgs, Int32 timeout)
at Skyline.DataMiner.Net.Connection.InternalHandleUnpackedMessages(DMSMessage[] msgs, Int32 timeout)
at Skyline.DataMiner.Net.Connection.HandleMessages(DMSMessage[] msgs, Int32 timeout)
at Skyline.DataMiner.Net.Connection.HandleMessages(DMSMessage[] msgs)
at Skyline.DataMiner.Net.Impersonating.ImpersonateConnection.HandleSingleResponseMessage(DMSMessage msg)
at Skyline.DataMiner.Library.Solutions.SRM.SrmManagersWithLog.RequestResponseEvent(Object sender, IManagerRequestResponseEventArgs e)
at Skyline.DataMiner.Net.IManager.Helper.IManagerEventRaiser.RaiseEvent[T](EventHandler`1 handler, T args)
at Skyline.DataMiner.Net.Messages.ResourceManagerHelper.SendAndReceiveMessage(IManagerRequestResponseEventArgs& args)
at Skyline.DataMiner.Net.Messages.ResourceManagerHelper.SetReservationInstancesOnServer(ReservationInstance[] reservationInstances, Boolean& success, Boolean isDelete, Object context, Boolean isSyncMessage, Boolean forceQuarantine, Boolean transactional, List`1 reservationsToIgnore, Boolean isGenerationForDefinition, Boolean validateSecurityViews)
at Skyline.DataMiner.Net.Messages.ResourceManagerHelper.InnerSetReservationInstances(ReservationInstance[] reservationInstances, Boolean& success, Boolean isDelete, Object context, Boolean isSyncMessage, Boolean forceQuarantine, Boolean transactional, Boolean validateSecurityViews)
at Skyline.DataMiner.Net.Messages.ResourceManagerHelper.<>c__DisplayClass119_0.<SetReservationInstances>b__0()
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()<—

at SRM.ResourceActions.Script.RemoveAction.RemoveResourceFromReservationInstance(ServiceReservationInstance reservationInstance, Node node)
at SRM.ResourceActions.Script.RemoveAction.RemoveFunctionResource()
at SRM.ResourceActions.Script.RemoveAction.ScriptFinished(Engine engine)
at SRM.ResourceActions.Script.Run(Engine engine)

Any help you can give on making sense of these errors would be greatly appreciated!
Ive Herreman [SLC] [DevOps Enabler] commented

Hi Philip,

The important parts of the error are:

SLDataGateway.API.Types.Exceptions.Queries.QueryException: Exception thrown while executing paged query:
Query(((ReservationInstance.End[DateTime] >=01/22/2026 15:28:26) AND (ReservationInstance.Start[DateTime] <=01/22/2026 15:28:26)) )

Elasticsearch.Net.UnexpectedElasticsearchClientException: The operation has timed out. —> System.Net.WebException: The operation has timed out.

This shows that the system was unable to fetch the required data from the database within a reasonable time.

Since the database is the root cause of the problem here, you'll have to take a closer look at the database logs (the elastic or opensearch logs) and the database server health metrics, like load, memory usage, etc.

If you need more assistance to investigate this, please reach out to support@dataminer.services