Skip to content
DataMiner DoJo

More results...

Generic selectors
Exact matches only
Search in title
Search in content
Post Type Selectors
Search in posts
Search in pages
Search in posts
Search in pages
Log in
Menu
  • Updates & Insights
  • Questions
  • Learning
    • E-learning Courses
    • Empower Replay: Limited Edition
    • Tutorials
    • Open Classroom Training
    • Certification
      • DataMiner Fundamentals
      • DataMiner Configurator
      • DataMiner Automation
      • Scripts & Connectors Developer: HTTP Basics
      • Scripts & Connectors Developer: SNMP Basics
      • Visual Overview – Level 1
      • Verify a certificate
    • Video Library
    • Books We Like
    • >> Go to DataMiner Docs
  • Expert Center
    • Solutions & Use Cases
      • Solutions
      • Use Case Library
    • Markets & Industries
      • Media production
      • Government & defense
      • Content distribution
      • Service providers
      • Partners
      • OSS/BSS
    • Agile
      • Agile Webspace
      • Everything Agile
        • The Agile Manifesto
        • Best Practices
        • Retro Recipes
      • Methodologies
        • The Scrum Framework
        • Kanban
        • Extreme Programming
      • Roles
        • The Product Owner
        • The Agile Coach
        • The Quality & UX Coach (QX)
    • DataMiner DevOps Professional Program
      • About the DevOps Program
      • DataMiner DevOps Support
  • Downloads
  • More
    • DataMiner Releases & Updates
    • Feature Suggestions
    • Climb the leaderboard!
    • Swag Shop
    • Contact
    • Global Feedback Survey
  • PARTNERS
    • All Partners
    • Technology Partners
    • Strategic Partner Program
    • Deal Registration
  • >> Go to dataminer.services

Is it possible to pause the DM SQLCleaningThread?

Solved1.10K views12th July 2023database MySQL
0
Jeff Douglass860 28th September 2022 0 Comments

I have many MySQL based production systems where the system and database has reached a size that is causing the SQLCleaningThread to constantly Fail/Timeout causing the database to continue to grow causing system performance and other database related issues. If this is not caught early it can result in the database containing hundreds of millions records that are outside the TTL window. When this happens I need to go into the system and manually cleanout the database via various methods which is very time consuming and can take many days to complete. One of my main problems when performing this manual cleanup procedure is that the DM SQLcleanup thread continues to run and fail hourly and when this process is initiated it can interfere my manual cleanup work.

Question: Is if there is a way to pause or stop the DM automatic SQL Cleanup process for a period of time while I am manually cleaning out the unneeded records from the database tables?

I am aware that in Cassandra I should not have these dbase cleanup issues but I have many systems cannot be immediately migrated to Cassandra so for the near future I will need to continue this  manually cleanup process to keep the systems functional.

Thanks in advance

Marieke Goethals [SLC] [DevOps Catalyst] Selected answer as best 12th July 2023

1 Answer

  • Active
  • Voted
  • Newest
  • Oldest
0
Jeroen Nietvelt [SLC] [DevOps Advocate]1.34K Posted 28th September 2022 5 Comments

Hi Jeff,

I believe there is a way to overwrite the default trending maintenance schedule. What we did in the past in these kind of situations is to edit the element.xml file of the impacted element(s) and append:

<Trending>
<TimeSpan1HourRecords unit=”hours”>43800</TimeSpan1HourRecords>
<TimeSpan5MinRecords unit=”hours”>43800</TimeSpan5MinRecords>
</Trending>

to the file. It should end up looking like this:

Note: Important is to not make any spelling mistakes as a “corrupt” element.xml file can cause the DataMiner agent not to start up anymore.

you can find this element.xml file on the DataMiner agent which is hosting the element(s), in the C:/Skyline DataMiner/Elements directory. There you will find a folder for each element hosted on that DataMiner agent. The folder will bear the same name as your element has.

Once edited, a DataMiner agent SW restart is required in order to have the DMA apply these custom settings on the element(s) in question.

After the DataMiner agent restart, you should see that the trend cleanup requests are no longer being executed for these elements. The corresponding runtime errors should therefore also no longer be generated.

It does mean that from this point on, these trend tables will only continue growing as DataMiner is no longer attempting to execute any cleanup actions. This means that the maintenance of these tables will need to be executed manually or by an external program.

Marieke Goethals [SLC] [DevOps Catalyst] Selected answer as best 12th July 2023
Jeff Douglass commented 28th September 2022

Thank you very much for this info. Does the entire DM need to be restarted or would restarting just the element work?

Thanks again

Jeroen Nietvelt [SLC] [DevOps Advocate] commented 28th September 2022

No problem Jeff, I’m afraid though that it will require a full DMA restart as DataMiner software only reads in the element.xml data once at startup and then keeps it in memory at runtime.

Jeroen Neyt [SLC] [DevOps Advocate] commented 29th September 2022

The configuration you suggested, states that for that specific element, 5 minute average trending data and 1 hour average trending data should be stored for 5 years. It surprises me that it would skip cleaning up the data if it’s configured like that. Could you elaborate a bit more on this configuration? Does it need to be exactly 5 years for this to work, or is 5 years and higher considered as the limit to be in need of a manual cleanup?

Jeroen Nietvelt [SLC] [DevOps Advocate] commented 29th September 2022

@Jeroen Neyt: In my context, the 5 years is considered as the limit to be in need of a manual cleanup. So yes i expect DataMiner to execute a cleanup action after 5 years.

Jeroen Neyt [SLC] [DevOps Advocate] commented 29th September 2022

Thanks for clearing that up. I thought you meant that the cleanup would not even be triggered at all. But it still gets triggered every hour, it only doesn’t do any actual delete queries for that specific element because there’s no data outside of that 5 year timeframe. Great suggestion!

Please login to be able to comment or post an answer.

My DevOps rank

DevOps Members get more insights on their profile page.

My user earnings

0 Dojo credits

Spend your credits in our swag shop.

0 Reputation points

Boost your reputation, climb the leaderboard.

Promo banner DataMiner DevOps Professiona Program
DataMiner Integration Studio (DIS)
Empower Katas
Privacy Policy • Terms & Conditions • Contact

© 2025 Skyline Communications. All rights reserved.

DOJO Q&A widget

Can't find what you need?

? Explore the Q&A DataMiner Docs