Are there any recommendations for hosting dataminer in the cloud?
Currently, we are looking at AWS and Azure.
This is what I have specified for a recent project: AWS Instance: m5d.4xlarge with Enhanced Networking › 100GB Dedicated OS C:/ drive on EBS Storage Volume should be on IO1 SSD Storage › 300GB Dedicated Cassandra D:/ drive on NVMe flash storage › 200GB Dedicated Elastic E:/ drive on NVMe flash storage › 100GB Dedicated Swap F:/ drive on NVMe flash storage › 100GB Dedicated Backup G:/ on EBS GP2 Storage* *This requirement will change in the future once Amazon FSx File Server and Storage is set up. This configuration utilizes low latency, high IOP NVMe ephemeral storage for the databases, but it is important to note, that if the instance (not the OS) is shutdown, then you will loos all data on the NVMe. This is also a risk should Amazon decide to shut down or move the instance for maintenance reasons. However, the difference in latency between IO1 SSD EBS storage and the NVMe is still very significant, the NVMe is the only option that mimics the latency and IOPS of SSD storage in a non-cloud environment. EBS GP2 and IO1 are both about 160ms of disk latency on RND4K tests. NVMe is 4ms.
Updated Recommendations:
These assume cassandra and elastic will co-located on the dataminer server.
Cloud Instance for Primary and Backup Server (Baseline Performance 100 Elements):
• Windows Server 2019
• AWS Instance: m5.4xlarge with Enhanced Networking (16 Core 64GB RAM)
• 300GB Initial Size Dedicated OS C:/ drive on EBS GP3 SSD Storage
• 300GB Initial Size Dedicated Cassandra D:/ drive on EBS GP3 storage
• 300GB Initial Size Dedicated Elastic E:/ drive on EBS GP3 storage
• 100GB Initial Size Dedicated Swap F:/ drive on EBS GP3 storage
• 600GB Initial Size Dedicated Backup G:/ on EBS GP2 Storage
› Regarding AWS Instances
Dataminer requires a static unchanging MAC address on network interfaces to facilitate server licensing. This can be provided through the Elastic Network Interfaces ENI Feature.
Cloud Instance for Primary and Backup Server (Maximum Performance Up to 250 Elements):
• Windows Server 2019
• AWS Instance: m5.8xlarge with Enhanced Networking (32 Core 128GB RAM)
• 300GB Initial Size Dedicated OS C:/ drive on EBS GP3 SSD Storage
• 300GB Initial Size Dedicated Cassandra D:/ drive on EBS GP3 storage
• 300GB Initial Size Dedicated Elastic E:/ drive on EBS GP3 storage
• 100GB Initial Size Dedicated Swap F:/ drive on EBS GP3 storage
• 600GB Initial Size Dedicated Backup G:/ on EBS GP2 Storage
› Regarding AWS Instances
Dataminer requires a static unchanging MAC address on network interfaces to facilitate server licensing. This can be provided through the Elastic Network Interfaces ENI Feature.
Dataminer server software supports the following operating systems:
› Windows Server Standard 2019
› Windows Server Standard 2022 (recommended)
Microsoft .NET Framework requirements:
Minimum:
Microsoft .NET Framework 3.5
Microsoft .NET Framework 4.6.2
Recommended:
We recommend to always upgrade to the latest .NET Framework version.
Other Microsoft software
• Microsoft SxS Source files must be accessible from the system for Server Roles and Features Installation
• Microsoft Visual C++ Redistributable Packages 2005 SP1, 2010 SP1 (x86 or x64) and 2015.
• Microsoft Web Services Enhancements (WSE) 2.0 SP3 for Microsoft .NET
• Microsoft Visio 2013, 2016, or 2019
3rd party software:
• WinPCap
• Ncap
• Wireshark
• Some kind of non-Java IP Scanner
• Crystal Diskmark
• Passmark
• Notepad++
• ManageEngine MIB Browser
• Postman
• Adobe Acrobat reader
NEW UPDATE:
Amazon has recently release a new storage option under EBS: IO2. This is a very high-performance, low latency SSD storage type that fully meets the most demanding requirements of a dataminer system.