There is a good article in the DM User Guide for this. I also remember having to do the following:
- the certificate needs to be valid for the FQDN (an exact match or valid for the domain the server is in)
- the server needs be configured with its hostname and DNS suffix.
- clients have to connect to the FQDN
- you can optionally force clients to connect the server by only allowing bindings to the FQDN (and prevent accessing via the server's IP address)
- There needs to be a DNS mapping between the FQDN and the IP address of the server. For testing purposes, you modify the 'hosts' file on a Windows client.
- At the time (2yrs ago), I also had to allow an HTTP binding on the localhost. I don't think there's a security issue with that, because this only allows the server to connect to itself via HTTP. This was needed for some functionality like rendering Visual Overview in a (legacy) dashboard.