Just checking in case there's some "outside-of-the-box" ideas from any one. (Hold on to your hat, we're diving deep)
Background:
Windows only supports file paths up to 260 characters. You can adjust a registry key and set windows OS to allow >260 characters. Unfortunately this doesn't actually make every application able to work with this.
Support for >260 characters requires source code changes for every single application. (things like, no longer able to use reference paths only absolute, needing to do special calls to do File and DirectoryIO instead of the usual, ...)
It's a significant change and that's even if it's at all possible to change (3rd party applications and API's where we cannot adjust the code ourselves and the like)
Especially in an environment like the CI/CD jenkins automation pipelines where you're dealing with over 40 different tools sometimes. (MSBuild, NuGet, SonarQube, DIS Validator, Major Change Checker, DCP API's, Cloud API's, All our internal stages and tools, ...)
We've somehow managed to either make sure paths stay barely under 260 or gotten most of our programs to work with >260 characters.
Not easy, because many programs put their default 'cache' folder locations and so on already >100 characters deep, before we even apply our own directory structure and filenames (e.g. before we changed the default location, NuGet was downloading things here: 136 characters C:\Windows\System32\config\systemprofile\AppData\Roaming\NuGet\packages\6da255cd87a32a8b910f79e20b7050a10efa5600$http_devcore3_81_nuget\[OUR FILE NAME]))
After a ton of tweaks with environment variables and configuration files everything more or less managed to pass through development, registration, CI/CD and so on.
The current problem:
But we're now, after all that, running into problems with the very last step of the 'pipeline', the dmapp installer on a user system itself.
The dmapp installer uploads the package by default to location:
C:\Skyline DataMiner\AppPackages\Uploads\[NameOfPackage_VersionOfPackage]_(32characterGuid)\[NameOfPackage_VersionOfPackage].dmapp"
For example: 147 characters
C:\Skyline DataMiner\AppPackages\Uploads\SRM.PLS.Nimbra.ITSInterface_v0.0.0-CU15_ea2b61811dff4a088665074539ca6ac2\SRM.PLS.Nimbra.ITSInterface.dmapp
The real issue is, that in order to UNZIP the .dmapp, all files inside the .dmapp need to have a path <260 characters as well or the unzipping doesn't work.
The default location for a dll out of a NuGet is:
AppInstallContent\Assemblies\ProtocolScripts\DllImport\[NameOfDll]\[Version]\lib\net462\[NameOfDll]
In total you can end up with the following path to a file inside a .dmapp that is 345 characters long:
C:\Skyline DataMiner\AppPackages\Uploads\SRM.PLS.Nimbra.ITSInterface_v0.0.0-CU15_ea2b61811dff4a088665074539ca6ac2\SRM.PLS.Nimbra.ITSInterface.dmapp\AppInstallContent\Assemblies\ProtocolScripts\DllImport\skyline.dataminer.communitylibrary.netinsight.nimbra\1.0.10325.8-1.0.0.x.8\lib\net462\Skyline.DataMiner.CommunityLibrary.Netinsight.Nimbra.dll
Now to fix this, we have several ideas but all of them are currently very unappealing so we're looking for anyone with a better idea:
- We try to rewrite the dmapp installer code so it can handle >260 path length.
- Problem: Every user computer would still need a registry edit to actually tell the OS to support it.
- Problem: This introduces a new and high Minimum Required DataMiner version, which reduces the value of all our work up to now by limiting what projects and users can use it.
- Problem: Uncertain if all the API's used in our code (e.g. for unzipping) support >260 characters. Might be out of our control once we start investigating our code base and could hit a dead-end.
- We reduce the default paths we have access on and set a restriction on library and solution names.
- Problem: Significant change to sourcecode, reduced readability.
- Problem: Reducing Skyline.DataMiner. into "SLC." as prefix (prefix is necessary for global uniqueness on nuget.org) would still (currently)only allow a 5-15 character name for a library. So "SLC.Nimbra" would be about the maximum currently.
- Problem: NuGet names and Solution names need to stay unique. Reducing the character count makes it much more likely to have conflicts.
- Problem: Reducing Library and Solution Names can impact readability for developers trying to find a piece of shared code to re-use using the search and NuGet Manager.
More details on the Path
The system developer decides the following (by defining names):
C:\Skyline DataMiner\AppPackages\Uploads\SRM.PLS.Nimbra.ITSInterface_v0.0.0-CU15_ea2b61811dff4a088665074539ca6ac2\SRM.PLS.Nimbra.ITSInterface.dmapp\AppInstallContent\Assemblies\ProtocolScripts\DllImport\skyline.dataminer.communitylibrary.netinsight.nimbra\1.0.10325.8-1.0.0.x.8\lib\net462\Skyline.DataMiner.CommunityLibrary.Netinsight.Nimbra.dll
with:
Automationscript solution name: SRM.PLS.Nimbra.ITSInterface
NuGet Library Name: Skyline.DataMiner.CommunityLibrary.Netinsight.Nimbra
CI/CD part of the code can adjust the following:
C:\Skyline DataMiner\AppPackages\Uploads\SRM.PLS.Nimbra.ITSInterface_v0.0.0-CU15_ea2b61811dff4a088665074539ca6ac2\SRM.PLS.Nimbra.ITSInterface.dmapp\AppInstallContent\Assemblies\ProtocolScripts\DllImport\skyline.dataminer.communitylibrary.netinsight.nimbra\1.0.10325.8-1.0.0.x.8\lib\net462\Skyline.DataMiner.CommunityLibrary.Netinsight.Nimbra.dll
The core software, dmapp installer adjusts the following:
C:\Skyline DataMiner\AppPackages\Uploads\SRM.PLS.Nimbra.ITSInterface_v0.0.0-CU15_ea2b61811dff4a088665074539ca6ac2\SRM.PLS.Nimbra.ITSInterface.dmapp\AppInstallContent\Assemblies\ProtocolScripts\DllImport\skyline.dataminer.communitylibrary.netinsight.nimbra\1.0.10325.8-1.0.0.x.8\lib\net462\Skyline.DataMiner.CommunityLibrary.Netinsight.Nimbra.dll
In case someone comes across this old question:
Eventually, we were able to fix every stage, 3rd party app and program used in the CI/CD except the above described dmapp installation step on user side.
The only way was with changes to the Core that caused a change in Minimum Support DataMiner.
The team made these changes and it's currently scheduled for possible release in DataMiner Version 10.2.10 (target date 23 Sept 2022)
So some created dmapp packages will unfortunately only be able to get installable on DataMiners from that version and up due to the windows max path length issue described above.
HI Jan,
Few ideas/proposals:
- core change: instead of working with 32-char GUID, it should also work with Base64 strings (22 chars).
- replace the name of your zip to 'app' and add a info.txt in the zip which has all the necessary info such as app name and app version etc..
Why? same behavior as protocols (protocol.xml)
- Unzipping itself, is this windows itself or a 3rd party tool such as Winzip, 7Zip or Winrar? Is it not possible not extract in a virtual directory? or define arguments that can be passed to the exe? Example: Instead of "Extract", use "Extract to".
- Unzipping/extracting in memory instead of in space.
Keep them in RAM is not an option? and from there onwards to drop them in another directory?
Hi Matthias,
The unzipping is also part of the core code, which mean mean changing the minimum supported dma version. We haven’t opened that sourcecode for now.
I’m still hoping we can avoid needing to apply our change in a DMA Version as it would reduce the value of our work up to now.
Unless we could release the ‘installer’ code versioned separated from the main DataMiner? If that could be released as its own little program that could be updated by users without having to update their entire setup, we might have more freedom to work with it.
Changing the name into App.dmapp might work but wouldn’t have enough impact on the path as a whole on its own. The core installer still adds a 32 character guid.
Though a combination of that trick together with changing the folder structure inside of the zip and adding a maximum to the user defined names is currently our best bet… unless someone can throw a different idea.