Hi, I wonder if there is documentation available detailing the file offload format (in particular, looking at time series data). As a user, I want to take the offload data, and convert it to specific storage format (in this case Apache Parquet files)
Thanks, Steven
Hi Steven,
In the following link (How DataMiner offloads data) you can find more information about the output format of the offload data. In essence, the file will contain comma separated values (CSV).
Hope it helps.
Do note however that using these files for the described use case (offload to a custom platform) is not ideal. Only MySql, MSSql and oracle are currently supported. Once the files have been offloaded to the configured database they are removed again from disk. They are being used internally by DataMiner as temporary storage and the format can even change in the future. As a workaround a dummy database could be configured to which the offload will never succeed. This would prevent that the files are deleted immediately.
To add to Tom’s comment, it’s possible to offload to file without the need to configure a dummy offload database, please see the Note section on https://docs.dataminer.services/user-guide/Advanced_Functionality/Databases/Optional_database_configurations/Offload_database.html
Thanks Miguel