time limits for SMTP-over-TLS

[Wietse Venema]

With Postfix 2.9 I'm hardening the Postfix SMTP server and client
implementations against exhaustion problems with byte-at-a-time
I/O, by enforcing a total time limit per SMTP line, instead of a
time limit per read(2) or write(2) system call.

This was already implemented in Postfix 2.8 postscreen. Without
time limits, an malicious client could tie up one postscreen session
for line_length_limit * postscreen_command_count_limit * smtpd_timeout
seconds. Even with stress-adaptive timeouts, this amounts to a
whopping 2048 * 20 * 10s = 4.7 days, which would not be acceptable.
With time limits, this reduces to postscreen_command_count_limit *
smtpd_timeout, or 200s with stress-adaptive timeouts.

I'm now adding similar deadline support to the Postfix TLS engine.
Below is a patch that naively implements read/write timeouts by
taking the per-line time limits for plaintext mail, and using the
same limits for TLS transactions.

This works fine for SMTP-over-TLS commands, as long as the sender
does not use overly-aggressive command pipelining; unfortunately
this approach is problematic for the DATA-over-TLS phase.

The issue is that with TLS, the DATA payload is sent in chunks of
up to 16 kbytes, unlike plaintext sessions, where the DATA payload
is typically sent in chunks of up to 1460-bytes. Given the way the
TLS protocol works, an entire 16 kbyte chunk must be received within
the time limit. Sending 16 kbyte takes a lot more time than sending
1460 bytes, and using the per-line time limit would certainly
penalize traffic across slow network links.

As you see, there is a lot of care for detail that goes into
maintaining Postfix.

Wietse

20110207

Cleanup: read/write deadline support for single_server TLS
applications (i.e. smtpd(8), smtp(8)). File: tls/tls_bio_ops.c.

diff --exclude=man --exclude=html --exclude=README_FILES --exclude=.indent.pro --exclude=Makefile.in -r -c /var/tmp/postfix-2.9-20110207/src/tls/tls_bio_ops.c ./src/tls/tls_bio_ops.c
*** /var/tmp/postfix-2.9-20110207/src/tls/tls_bio_ops.c Fri Dec 17 19:40:59 2010
--- ./src/tls/tls_bio_ops.c Tue Feb 8 19:19:00 2011
***************
*** 103,108 ****
--- 103,121 ----
/* System library. */

#include <sys_defs.h>
+ #include <sys/time.h>
+
+ #ifndef timersub
+ /* res = a - b */
+ #define timersub(a, b, res) do { \
+ (res)->tv_sec = (a)->tv_sec - (b)->tv_sec; \
+ (res)->tv_usec = (a)->tv_usec - (b)->tv_usec; \
+ if ((res)->tv_usec < 0) { \
+ (res)->tv_sec--; \
+ (res)->tv_usec += 1000000; \
+ } \
+ } while (0)
+ #endif

#ifdef USE_TLS

***************
*** 129,134 ****
--- 142,161 ----
int err;
int retval = 0;
int done;
+ struct timeval time_limit; /* initial time limit */
+ struct timeval time_left; /* amount of time left */
+ struct timeval time_entry; /* time of tls_bio() entry */
+ struct timeval time_now; /* time after SSL_mumble() call */
+ struct timeval time_elapsed; /* total elapsed time */
+
+ /*
+ * Deadline management is simpler than with VSTREAMs, because we don't
+ * need to decrement a per-stream time limit. We just work within the
+ * budget that is available for this tls_bio() call.
+ */
+ time_limit.tv_sec = timeout;
+ time_limit.tv_usec = 0;
+ GETTIMEOFDAY(&time_entry);

/*
* If necessary, retry the SSL handshake or read/write operation after
***************
*** 194,205 ****
done = 1;
break;
case SSL_ERROR_WANT_WRITE:
- if (write_wait(fd, timeout) < 0)
- return (-1); /* timeout error */
- break;
case SSL_ERROR_WANT_READ:
! if (read_wait(fd, timeout) < 0)
! return (-1); /* timeout error */
break;

/*
--- 221,242 ----
done = 1;
break;
case SSL_ERROR_WANT_WRITE:
case SSL_ERROR_WANT_READ:
! GETTIMEOFDAY(&time_now);
! timersub(&time_now, &time_entry, &time_elapsed);
! timersub(&time_limit, &time_elapsed, &time_left);
! timeout = time_left.tv_sec + (time_left.tv_usec > 0);
! if (timeout <= 0) {
! errno = ETIMEDOUT;
! return (-1);
! }
! if (err == SSL_ERROR_WANT_WRITE) {
! if (write_wait(fd, timeout) < 0)
! return (-1); /* timeout error */
! } else {
! if (read_wait(fd, timeout) < 0)
! return (-1); /* timeout error */
! }
break;

/*

[Wietse Venema]

This post might be inappropriate. Click to display it.

[Victor Duchovni]

Post by Wietse Venema

Post by Wietse Venema
This works fine for SMTP-over-TLS commands, as long as the sender
does not use overly-aggressive command pipelining; unfortunately
this approach is problematic for the DATA-over-TLS phase.

The problem being that the "per SMTP line" time limits become "per
TLS transaction" time limits. While SMTP plaintext is sent in small
chunks of up to 1-2 kbytes, TLS transactions can be as large as 16
kbytes, and an entire TLS transaction must be received before the
plaintext can be made available to the application.
This is only a problem when time limits are set really short with
Postfix SMTP server stress mode, and connections are slower than
1 kbyte/s or so. The Postfix SMTP client sends chunks of up to 4
kbytes over ethernet-like networks which makes the problem with
large transactions less of an issue.

With the default stress-dependent timeout of 10s, indeed each SSL_read()
needs to succeed in under 10s, so senders that:

- Actually use maximum-sized 16k buffers above the SSL_write()
layer.

- Have trouble delivering 128Kbits in 10s.

may get dropped under stress conditions. Now 12.8 Kbps is a very modest
bandwidth these days, and is only the limit under stress conditions, I
don't think we need to worry too much about setting the bar this high.
Receiving systems that have clients that need a bit of extra help can
raise the stress timeout...

Post by Wietse Venema
It seems that the best approach here is to try the new per-transaction
time limit as the default mechanism during the Postfix 2.9 development
cycle, with the option to use the traditional per-read and per-write
time limit where interoperability over slow connections is more
important than availability.

Do we really need a switch, or can we just document raising the stress
timeout as the work-around?