What's the best way to check if an SMTP server is SSL-enabled or not?

That depends whether you mean SSL or TLS.

SSL has it's own dedicated port at TCP/465. The best way to test for it's presence would be to use OpenSSL's wonderful s_client which will negotiate the SSL trickery for you.
```
openssl s_client -connect localhost:465
```
If your server isn't bound to localhost then obviously replace that with the IP or hostname.
TLS looks just like normal SMTP at first. The encryption is negotiated from and on-top of the plain-text protocol. You can test whether it is available by issuing an EHLO request to the server. You can use Netcat or Telnet clients for this.
```
$ nc -v localhost 25
localhost [127.0.0.1] 25 (smtp) open
220 mail.example.com ESMTP Exim 4.69 Fri, 11 Sep 2009 09:25:20 +0100
ehlo test
250-mail.example.com Hello localhost [127.0.0.1]
250-SIZE 10485760
250-PIPELINING
250-STARTTLS
250 HELP
```
The important line is second from last which advertises the STARTTLS capability.

In order to say how to enable SSL/TLS for your mail server you'll need to tell us what mail package you're using.

Related