Hello Dojo,
Currently dealing with a legacy bespoke routine where the file size of an export has increased over time to support new requirements - therefore we should no longer export to "Documents" as that folder, by design, expects files to be less than 20MB.
When file synchronization of files fails, due to the file size being more than 20MB, we get messages like
DMA name --> Error during synchronization --> File "ABC.zip" is more than 20MB --> Sync is ignored
Hence the question: will the sync start and then fail ("during") before being "ignored"?
Or is the error triggered just after a file size check, hence the sync process will ignore the files before attempting to sync the first few MB?
We're working with our support squad to re-tune the routine and export to a different path, out of Documents. Any steer on other mitigation steps we can take will be helpful.
We can remove the notice messages, but before addressing the effect, I thought to check on the community about wider considerations on the cause.
Most CUBE clients seem to become very slow when this happens as many more lines need to be polled
(files will fail sync on all the DMAs of the cluster, as shown above) and most servers normally used by operators have no eventing configuration.
PS: Will the sync run only at configured times (e.g. midnight sync) or is it triggered also on file creation in the Documents folder?
Hi Alberto,
A size check is being done before synchronizing the file, so the file will be ignored completely and not partially synced.
The sync will run both at file creation and a midnight sync normally.
Note that this size can be configured in the MaintenanceSettings.xml file:
https://docs.dataminer.services/user-guide/Reference/Skyline_DataMiner_Folder/More_information_on_certain_files_and_folders/MaintenanceSettings_xml.html?q=maintenancesettings%20maxsize#documentsmaxsize
Increasing this to a size like 500MB could cause potential issues like out of memory exceptions in SLDMS, note that SLDMS still is a 32 bit process and thus has a limitation of maximum 4GB of memory until it is eventually ported to 64 bit.
Everything that needs to be synced needs to be sent over the wire and is loaded in memory in SLDMS before being written to disk on both the sending and receiving agents.
Therefore I would not advise setting this to 500 MB, 50MB is the max I would set it to, or as you mentioned inspect the documents that are synced and see if they can be split up or reduced in size.
Again, thanks for your feedback, Dieter
We’re likely to leave size settings unchanged so we don’t build any more load on SLDMS as advised – good shout!
Not clear why the simple presence of a file exceeding the size limits triggers the a notice message on all the DMAs in the cluster: this happens at every sync after file creation, so there seems to be no memory of the files that sync will need to ignore – but as you say, at this stage it might be helpful for us to inspect the documents and evaluate how to reduce the size.
The reason why this notice is seemingly happening on all agents in the cluster is because the notice is generated on the agent that wants to sync with the origin of the document, so it is generated on all agents except for the one where the file originates from.
Thanks for the prompt feedback Dieter – if the default is 20MB, given there’s space on all drives of the DMAs, could this become ~max 500MB?
Or would that incur other limitations & knock-on effects?