Just in case anybody hits the same problem I did…
I have a VPS set up, partly to provide email services for a few people. I use exim with SMTP AUTH over TLS for mail submission. This was working well with Windows Mail and Thunderbird, but OS X Leopard Mail.app wasn't working. It got as far as sending the STARTTLS SMTP command, but then the connection closed with errors in Exim's mainlog like:2008-12-29 20:38:03 TLS error on connection from my-client [my-ip] (gnutls_handshake): A TLS packet with unexpected length was received
or when I tried OpenSSL rather than GnuTLS2008-12-28 22:13:12 TLS error on connection from my-client ([192.168.0.3]) [my-ip] (SSL_accept): error:00000000:lib(0):func(0):reason(0)
It made no difference whether I was trying to connect on port 25, 465 or 587.
It turns out the problem was my self-signed certificate. Mail.app was didn't like it, and even when I clicked "Connect" at the warning dialogue it would just close the connection without any further diagnostics (though it was fine with the cert for IMAP). Taking more care with the certificate dialogue and telling Mail.app to always trust the certificate caused it to suddenly spring into life.