Saturday, February 28, 2009

Making Exim4 send outgoing mail through Gmail

Say you have a personal server on the Internet, and you want it to send e-mail.

In the good old days, that meant nothing more than ensuring you could make outbound connections on tcp port 25. In our world of pervasive junk e-mail, however, it can take a great deal of work to get your sent mail to comply with all of the guidelines that major e-mail providers recommend to avoid having your mail classified as spam (if it is, it will likely never be seen). You'll get your own domain in DNS so you can set up DKIM milters and SPF records, make sure your MTA has appropriate retry rules as expected by greylisting hosts and that it responds properly to callback verifications, get a static IP or two in a reputable neighborhood - oh, and make sure your ISP allows you to control the PTR record. You might even throw in some Hashcash headers for good measure. And even after all that, you might have a hard time getting your mail through to your recipients until you've been sending from that IP for long enough to build up a digital reputation.

Hardly seems worth it anymore, does it?

One potential solution (although one I haven't seen discussed or used much) is to teach your server how to authenticate to an existing mail provider so it can send mail through it. Your provider does all the work to ensure availability and deliverability, so you don't have to worry about it. And there are some good free e-mail providers.

Here is the solution I rolled for a particular server. I configured exim4 to send e-mail using an account on a Google Apps domain (so, essentially, it's going through Gmail). Gmail provides a free SMTP sending interface, and one thing Exim is good at is speaking SMTP, but it takes a bit of configuration to express what we need.

The centerpiece of my approach is a file called in the Exim config directory. It maps addresses of Gmail/Google Apps accounts to passwords. When Exim gets an outgoing mail with a sender address in the map, it uses the password to authenticate to Gmail's SMTP server so Gmail can resend it.

A sample         S00perSekritP@ssw0rd j2K&.sh4J em8baiXu

First, I defined a router called use_gmail:

debug_print = "R: use_gmail for $domain (sender $sender_address)"
driver = manualroute
require_files = CONFDIR/
senders = lsearch;CONFDIR/
route_list = *
transport = smtp_through_gmail

The manualroute router type is for manually routing (that is, determining the next-hop SMTP server) based on certain tests. Normally you make the decision based on the destination address, but here we check the sender address. And the manual destination we set for those addresses is always, port 587.

It instructs that matching mail be sent to its destination using the smtp_through_gmail transport:

debug_print = "T: smtp_through_gmail for $local_part@$domain"
driver = smtp
connection_max_messages = 1
hosts_require_auth = *
hosts_require_tls = *
tls_certificate = CONFDIR/exim.crt
tls_privatekey = CONFDIR/exim.key
tls_verify_certificates = /etc/ssl/certs/ca-certificates.crt

Our transport always uses SMTP, and the remote end always requires TLS and authentication. We verify the certificate presented by the remote end ( to make sure the certificate is signed by an authority in our local list, and that it actually is a certificate for the right address. The exim.crt and exim.key files are for our client-side certificate. Creating such a key/cert pair is beyond the scope here, but there is good info all over the net.

One other parameter there which bears explanation is connection_max_messages = 1. It says that only one message will ever be sent per connection. I'm not completely sure whether that's necessary, but I suspect that if it were not there, Exim might try to send multiple messages from different senders after authenticating as only one. That would be bad- you'd get messages coming from the wrong sender all over the place.

The last step is to tell the authenticators how to get the appropriate passwords. Since I didn't need to use the default login or plain authenticators, I just tweaked them for my needs. If you have a separate need for them (for example, if you need to want to allow incoming mail also authenticated with username/password pairs) then you'll need to work out how to have those authenticators get the right passwords from the right place for each situation.

Here are my authenticators, though:


driver = plaintext
public_name = PLAIN
client_send = "<; ${if !eq{$tls_cipher}{}\

driver = plaintext
public_name = LOGIN

# Return empty string if not non-TLS AND looking up
# $host in passwd-file yields a non-empty string;
# fail otherwise.
client_send = "<; ${if and{\
; ${extract{1}{::}{PASSWDLINE}}\
; ${sg{PASSWDLINE}{\\N([^:]+:)(.*)\\N}{\\$2}}"

I would go through an explanation of all that but it makes my head hurt just to look at it. It just says only to use passwords when the connection is encrypted (a tls_cipher is in use) and to get them from the line. It also tells Exim the details of how both authentication mechanisms (PLAIN and LOGIN, they're pretty standard) work. But ugh. Even Perl would be easier to read than Exim's funkotron string expansion language. If you need to find out more, check the Exim docs here and here.

That's all the necessary configuration. You'll probably want to make a few extra tweaks, though. For example, standard Exim installations don't allow normal users to set the envelope (sender) address to something besides user@mailname. On my system, all the user accounts are trusted to send mail from any of the configured addresses, so I just add them all to Exim's trusted_users list. There are other ways to provide that permission in a more fine-grained way, but again, beyond the scope here.

To set the sender address explicitly for a mail, pipe the mail (including all headers) into

/usr/sbin/sendmail -oi -f $SENDER_ADDRESS <list of recipient addresses>

(Change the path to your sendmail binary as necessary.)

And that's it. Outgoing mail from this server having one of the supported sender addresses doesn't ever get lost in spam folders anymore.

I am no kind of Exim expert, so I might have missed some ways to make this easier, more general, or more robust (although I haven't had any problems yet). Comments and improvements are welcome.

Tuesday, February 24, 2009

broken anonymous comments

Sorry to those who tried to comment anonymously on the safelite post. Apparently the word verificationery is broke. I disabled word verification for comments on this blog until they fix it.

Monday, February 23, 2009

safelite exploit

Tav recently issued a challenge to get around the security of a module he made, intended to create a sort of restricted execution sandbox in the Python interpreter. Apparently if it can be made secure enough after lots of people look at it, then it could get included in the Python standard library.

That seems like a worthwhile thing to have available, to make uses like the Google App Engine more easily implemented and secured. It works (at least, the current version does) by trying to hide anything you could use from code to write to the local filesystem. You get a replacement function for file()/open() which only allows opening in read mode, and __import__(), execfile(), reload(), etc, are taken out of the builtins dictionary. So (hopefully) the user can not import anything at all.

To go further than that, tav removes the func_* attributes from the FunctionType dictionary- if you had func_closure, as an example, you would be able to inspect the closure attached to the replacement open() function and get at the real one. (My recipe on ASPN shows you one way to do that.) A few other attributes are removed from the dictionaries of default types, like gi_code on GeneratorType.

The replacement open() function is (now) carefully coded to avoid trusting any globals or the contents of the __builtins__ dictionary when it is run- otherwise, you'd be able to trick it into acting in different ways when it uses values from there.

I came up with an exploit which hinges on the continued presence of the compile() builtin in the sandbox. When you use compile(), you get a code object out. When you have a code object, you can create new code objects (using type(code_object)(arguments)). Since you can come up with arbitrary bytecode to put in a new code object, you can make the code object thus created do a few things that you can't do in normal python. The most useful one in this case is that you can get access to the traceback object from an exception without sys.exc_info() or sys.exc_traceback.

I won't go too far into details on that, except to say that when you get into an exception handler, the stack is topped with the exception object, as well as the traceback and type objects you'd get from sys.exc_info(). Roll a little custom bytecode, and you can store it off of the stack rather than throwing it away (which is what the compiler will usually do):

>>> f = type(lambda: 0)(type(compile('1', 'b', 'eval'
))(2, 2, 4, 67, 'y\x08\x00t\x00\x00\x01Wn\x09\x00\x01'
(None,), ('stuff',), ('g', 'x'), 'q', 'f', 1, ''),
globals(), None, (TypeError,))
>>> dis.dis(f)
1 0 SETUP_EXCEPT 8 (to 11)
3 LOAD_GLOBAL 0 (stuff)
8 JUMP_FORWARD 9 (to 20)
>> 11 POP_TOP
13 STORE_GLOBAL 0 (stuff)
16 JUMP_FORWARD 1 (to 20)
>> 20 LOAD_FAST 1 (x)
23 LOAD_FAST 0 (g)

That function returns TypeError, because I need to get it to run underneath the replacement open() function, and the easiest way (there are others) is to overload TypeError- the only global that it references.

__builtins__.TypeError = f

I just call the replacement open() with a mode parameter of 2, so that it will load and call TypeError:

FileReader('foo', 2)

..whereupon it has kindly stored the traceback object in the global dict under the name "stuff". Traceback objects contain references to frame objects, and you can follow frame objects up the call chain, and so we can easily get to the frame containing the replacement open() function:


From that point, it's trivial to pull out the real open() function from the local variables of the frame, and use it:

('w00t', 'w').write('yay\n')

That works in Python 2.4 through 2.6, and probably some earlier 2.x versions. I seriously doubt it works in 3.0, but I haven't tried, and it might be adaptable to work there.

What's the right approach to fixing this hole? I'm not sure. If tav decides to disallow compile(), I haven't yet found any other way to get at a code object, so that would plug it up as far as I know. On the other hand, it would be real nice to be able to keep compile(). Give the restricted-environment users as much power as possible without sacrificing security. If it is possible to remove f_back or f_locals or tb_frame from their respective builtin-type dictionaries, that would plug the hole, but would probably break the reporting and display of normal exception tracebacks.

Maybe Python can be made not to provide the traceback object in an exception handler's stack frame- I admit I don't even know why it does. Does it support some old, no-longer-documented syntax? That would plug up this hole without sacrificing any functionality that I know of, but there might remain some related exploits.