Improving STARTTLS authentication in the Sendmail MTA

After having worked on what I think is the right way to setup X.509 certificates for use in SMTP STARTTLS transactions , I am continuing my work on how we can improve the way the Sendmail MTA handles STARTTLS authentication. This article explains how the Sendmail MTA handles this and what the limitations are. We will  finally make some suggestions on how we could improve our implementation, and of course your comments to this article are not only welcome, but they will be particularly useful.
Opportunistic TLS or mandatory TLS?
In most cases, mail platforms use opportunistic TLS, sometimes even without the mail administrator knowing this. For example a default setup of the Sendmail Opensource MTA is likely to use opportunistic TLS as a client (when sending mail), and you don’t need to do anything for this. Opportunistic TLS brings encryption of your communication up to the next hop. It does not provide any form of authentication: you may still be sending your mail to the wrong server, just that the communication will be encrypted!
On the other hand, mandatory TLS is when you need to ensure you’re speaking to the right server, and this works both ways: inbound and outbound. Policy configuration will ensure that you speak to the right server, i.e. when the MTA has decided to send a message to, say, the host relay.wolfhugel.eu, you want to make sure that this host shows a X.509 certificate which is adequate and which you trust.
Mandatory TLS makes sense from the moment you need to be sure who you’re talking to, or who is connecting to your mail server. This policy enforcement can be partial: most real life situations use standard Internet mail routing (MX records), eventually opportunistic TLS, and mandatory TLS to/from just a few remote sites.
Current implementation in the Sendmail MTA
Thanks to the use of rulesets to handle TLS enforcement, site administrators have a vast choice of what they can do with respect to validating a TLS connection. The only limitations are:

• The information coming from the certificate and provided by the MTA to the ruleset engine
• The ability of the rulesets to do things (and this is quite a large set of things).

Most users will even not use these rules and prefer the existing methods which are part of the MTA package, mostly by ways of using entries in the “access_db” file.
The TLS_Srv and TLS_Clt tags are used respectivement when the Sendmail MTA sends (TLS_Srv) or receives (TLS_Clt) mail and it allows defining one or several actions. These filters apply when selecting the actual name of the server we’re talking to (or the domain of it), this means whatever appears in the MX. For fining control based on the recipient’s address there is a TLS_Rcpt keyword which is applied per recipient, regardless of the server we’re talking to. There is no such rule, sender address based, for incoming mail.
A few examples:

TLS_Srv:wolfhugel.eu      VERIFY:128+CN
TLS_Rcpt:wolfhugel.eu     VERIFY:128+CN++CI:/C=GB/.../O=MyCA

The first line means that when talking to any server called “wolfhugel.eu” or within this domain (mail.wolfhugel.eu, relay.wolfhugel.eu, …), we want the TLS connection to be shown with a certificate whose Certification Authority is trusted by our system, with at least a 128 bit encryption algorithm and in addition we will check the CN of the presented certificate to ensure it corresponds to the one we’re actually talking to. So if our MX record is “relay.wolfhugel.eu” and we send a mail to this server, we want the certificate to be trusted and to have “relay.wolfhugel.eu” in its CN field.
The second line applies to any mail going to someone within the wolfhugel.eu domain, regardless of the mail server we’re actually sending the mail to. We do all verifications described above and in addition we want that the Certificate Issuer (this can be an intermediate certification authority) has a Subject line of “/C=GB/…/O=MyCA”, i.e. we indicate clearly which CA we trust for this domain.
Full documentation is provided in the Opensource Sendmail MTA package in the cf/README file.
Another use of TLS authentication is to permit, or not, relaying, i.e. the goal is to say that if you show a given certificate you may use this SMTP server to send mail anywhere (typically you would use the MSA submission port, 587/tcp). I do use this when I’m roaming: regardless of the IP address I show, my home MTA with allow me to send messages to anyone if I can prove, by ways of the adequate certificate, it’s me.
The verification is again done in two steps by using the “access_db” map: first we check if the certificate issuer is authorized, and then eventually we check the certificate itself, like in following example:

CertIssuer:/C=GB/.../O=MyCA        RELAY
CertIssuer:/C=GB/.../O=AnotherCA   SUBJECT
CertSubject:/C=GB/.../CN=CWOLF     RELAY

The first line means that we will trust anyone showing a certificate issued by the certification authority “/C=GB/…/O=MyCA” (which again can be an intermediate authority). The second line says that if a certificate is presented signed by “/C=GB/…/O=AnotherCA” then we’ll check the Subject of the certificate itself, and we will authorize any listed entry with the adequate “CertSubject” entry.
Anything else? Just write your own rules, and this article won’t go into this. You can write any rules using the “sendmail.cf” language and the values provided for certificates. Following macros are available (taken from the documentation):

• ${cert_issuer} holds the DN of the CA (the cert issuer).
• ${cert_subject} holds the DN of the cert (called the cert subject).
• ${cn_issuer} holds the CN of the CA (the cert issuer).
• ${cn_subject} holds the CN of the cert (called the cert subject).
• ${tls_version} the TLS/SSL version used for the connection, e.g., TLSv1, TLSv1/SSLv3, SSLv3, SSLv2.
• ${cipher} the cipher used for the connection, e.g., EDH-DSS-DES-CBC3-SHA, EDH-RSA-DES-CBC-SHA, DES-CBC-MD5, DES-CBC3-SHA.
• ${cipher_bits} the keylength (in bits) of the symmetric encryption algorithm used for the connection.
• ${verify} holds the result of the verification of the presented cert.
  Possible values are:

  OK	verification succeeded.
  NO	no cert presented.
  NOT	no cert requested.
  FAIL	cert presented but could not be verified, e.g., the cert of the signing CA is missing.
  NONE	STARTTLS has not been performed.
  TEMP	temporary error occurred.
  PROTOCOL	protocol error occurred (SMTP level).
  SOFTWARE	STARTTLS handshake failed.

• ${server_name} the name of the server of the current outgoing SMTP connection.
• ${server_addr} the address of the server of the current outgoing SMTP connection.

Limitations in the current implementation
The provided primitives already allow a lot of flexibility, but I feel there are a few limitations which are worth being discussed and which could lead to improvements.
An important limitation I came across when setting up TLS recently is that the Sendmail MTA does not handle the X509v3 “Subject Alternative Name” extension which allows specifying more than one host name in a certificate: my certificate could have a CN=tea.wolfhugel.eu and the extension would list this and all other names I do use the certificate for (“DNS:mail.wolfhugel.eu”, “DNS:relay.wolfhugel.eu”, …). Also one could argue that when doing name matching in certificate validation the current rules do not support the use of wildcard certificates (i.e. “CN=*.wolfhugel.eu”, “DNS:*.wolfhugel.eu”).
It seems like the Sendmail MTA it is somewhat difficult to define rules with intermediary certification authorities when trusting or accepting certificates.
Some proposals to improve the current handling in Sendmail MTA

Here are a few proposals to hopefully improve the way the Sendmail MTA uses certificates in STARTTLS connections and how to decide about giving, or not, authorization to connect or to relay. Comments on this are more than welcome, as users’ feedback is essential for improving the product!
Implement X509v3 Subject Alternative Names
This is a significant change and it requires source code changes in the MTA, but I feel it is probably the most needed extension: the Sendmail MTA would extract the DNS Subject Alternative Names from the presented certificates and expose them to the configuration file in macros $&{altname_subject} and $&{altname_issuer}. These macros can later be used in any rule where they are available for deciding to give, or not, access to the SMTP service.
Create new rules to use the Subject Alternative Names
The two new macros defined with the new names extracted from the certificates can be used to extend the current way the MTA gives authorization. Here is a proposal to extend the current rules.
For the TLS_Srv/TLS_Rcpt entries in the access DB we would add a new attribute called “CA” (as Cert Altnames) who behaves identically to the “CN” attribute but would mean match the CN or any alternative name, and the example given in the beginning of this article would become:

TLS_Srv:wolfhugel.eu      VERIFY:128+CA
TLS_Rcpt:wolfhugel.eu     VERIFY:128+CA++CI:/C=GB/.../O=MyCA

This would mean the the host we’re connecting to must be found either in the CN of the certificate or in any DNS value of the alternative names for the verification to be successful.
Similarly when authorizing a remote party to relay, with the CertIssuer/CertSubject tags, we would add a new value to the access database: “ALTNAMES:xxx” would create an indirection telling to search for a key called “SubjAltName:xxx:value” in the access map which in turn would authorize or prevent access as shown in this example:

CertIssuer:/C=GB/.../O=MyCA         ALTNAMES:SMI
SubjAltName:SMI:cwolf@sendmail.com  RELAY

The meaning of the above would be that whenever a connection is made with a certificate issues by “/C=GB/…/O=MyCA” we will be checking the alternative names and our key for the lookup is “SMI”. If any of the alternative names is “cwolf@sendmail.com” relaying will be authorized, and yes here we do allow Email addresses as alternative names.
Implement the wildcard certificates
Another point which has not been implemented in supporting wildcard certificates, i.e. a certificate whose CN or alternative name contains a star, like “*.wolfhugel.eu” meaning this certificate covers any host within the “wolfhugel.eu” domain name. This implementation would be transparent, i.e. no changes in the access map would be required.