Per-Recipient Data Responses

Post by Claus Assmann
[resending with correct From: address]

#define SMFIP_PRDR 0x00200000L /* MTA supports PRDR */
isn't really necessary as the milter must parse the MAIL arguments
anyway to determine whether the client uses PRDR. If the MTA doesn't
support it, it doesn't send the MAIL command with a PRDR argument
to the milter anyway because it rejects that as unknown parameter.
Hence I'm not sure whether it's worth "wasting" a bit on it.

What are the odds that an MTA passes MAIL FROM..PRDR to the Milter,
yet is unwilling to receive the new multi-reply response from the
Milter?

SMFIR_PRDR (one reply per recipient, each formatted as with
SMFIR_REPLYCODE). If we can freeze the details, then it could be
in Postfix 2.9, which is due early 2012.

int
smfi_setreplies(SMFICTX *ctx, unsigned int nreplies, int *rcodes, const char **rtexts)
where
nreplies: number of replies, this must match the number of valid
RCPTs for the transaction.
rcodes: a status array of size nreplies, each element must be one of
SMFIS_TEMPFAIL
SMFIS_REJECT
SMFIS_ACCEPT
rtexts: a text array of size nreplies, each element must be either
NULL or a valid SMTP reply text which matches the corresponding rcode,
i.e.,
SMFIS_TEMPFAIL 4xy
SMFIS_REJECT 5xy
SMFIS_ACCEPT 250
including enhanced status code and a textual description, ending
in \r\n. The text must be formatted as described in the SMTP RFCs,
i.e., 821/2821/5321.

What would you recommend for the (Sendmail Milter protocol) wire format?

pkt-length command nreplies rcodes*nreplies null-terminated-rtext*nreplies

All this would have to fit in the max packet length which is 65535
if I recall correctly, or is that applicable only for body chunks?

Will there be a problem with local RCPT TO counts > the minimum 100?
Postfix currently has no explicit upper limit, i.e. INT_MAX. The PRDR draft
cites RFC2821 without excluding multi-line replies.

Wietse

Claus Assmann

2011-11-25 16:45:00 UTC

Post by Claus Assmann
#define SMFIP_PRDR 0x00200000L /* MTA supports PRDR */
isn't really necessary as the milter must parse the MAIL arguments
anyway to determine whether the client uses PRDR. If the MTA doesn't

What are the odds that an MTA passes MAIL FROM..PRDR to the Milter,
yet is unwilling to receive the new multi-reply response from the
Milter?

Probably > 0, so we should add that flag.

Post by Wietse Venema
What would you recommend for the (Sendmail Milter protocol) wire format?

That's the ugly part I didn't want to think about yet... only after
we come to some kind of agreement on the API.

Post by Wietse Venema
pkt-length command nreplies rcodes*nreplies null-terminated-rtext*nreplies

I'm wondering whether there should be a length parameter for each
null-terminated-rtext (to allow easy allocation of per-RCPT reply
buffers plus validation of the packet) but that's not used elsewhere
in the protocol, so we should go with your suggestion.

#define SMFIR_REPLIES 'z' /* reply for PRDR */
could be used for the command.

Post by Wietse Venema
All this would have to fit in the max packet length which is 65535
if I recall correctly, or is that applicable only for body chunks?

It's applicable to all packets. However, _FFR_MDS_NEGOTIATE can be
used to increase that size to 256K or even 1M.

Post by Wietse Venema
Will there be a problem with local RCPT TO counts > the minimum 100?

If the packet size is increased to 1M then it seems unlikely to run
into that limit, but it certainly needs to be mentioned in the doc
as a restriction.

Wietse Venema

2011-11-25 17:20:18 UTC

What are the odds that an MTA passes MAIL FROM..PRDR to the Milter,
yet is unwilling to receive the new multi-reply response from the
Milter?

Probably > 0, so we should add that flag.

Post by Wietse Venema
What would you recommend for the (Sendmail Milter protocol) wire format?

That's the ugly part I didn't want to think about yet... only after
we come to some kind of agreement on the API.

Post by Wietse Venema
pkt-length command nreplies rcodes*nreplies null-terminated-rtext*nreplies

Actually we can forget about the rcodes*nreplies part.

The rcode isn't sent with SMFIR_REPLY, so it should not be sent
with SMFIR_REPLIES either. With SMFIR_REPLY all three reply arguments
are inside the rtext.

That makes it:

pkt-length command nreplies null-terminated-rtext*nreplies

Where nreplies can be 32-bit. With the current 1M reply size limit,
sending 65536 replies would already be challenging, but numeric
limits are easy to change, whereas protocol field widths are not.

Otherwise, what you write sounds good to me. I agree we don't need
to introduce a new on-the-wire array encapsulation just to send an
array of RCPT TO replies.

Wietse

Post by Claus Assmann
I'm wondering whether there should be a length parameter for each
null-terminated-rtext (to allow easy allocation of per-RCPT reply
buffers plus validation of the packet) but that's not used elsewhere
in the protocol, so we should go with your suggestion.
#define SMFIR_REPLIES 'z' /* reply for PRDR */
could be used for the command.

Post by Wietse Venema
All this would have to fit in the max packet length which is 65535
if I recall correctly, or is that applicable only for body chunks?

It's applicable to all packets. However, _FFR_MDS_NEGOTIATE can be
used to increase that size to 256K or even 1M.

Post by Wietse Venema
Will there be a problem with local RCPT TO counts > the minimum 100?

If the packet size is increased to 1M then it seems unlikely to run
into that limit, but it certainly needs to be mentioned in the doc
as a restriction.

Claus Assmann

2011-11-25 17:46:14 UTC

Post by Wietse Venema
pkt-length command nreplies rcodes*nreplies null-terminated-rtext*nreplies

Yes, but the API allows to omit the text:

! int
! smfi_setreplies(SMFICTX *ctx, unsigned int nreplies, int *rcodes, const char **rtexts)
! rcodes: a status array of size nreplies, each element must be one of
! SMFIS_TEMPFAIL, SMFIS_REJECT, SMFIS_ACCEPT

! rtexts: a text array of size nreplies, each element must be either
! NULL or a valid SMTP reply text which matches the corresponding rcode,
[1]

The existing API uses a return value for the xxfi_*() functions
plus a separate "custom reply" that can be set via smfi_setreply()[2],
while this function basically merges the two (reply code and reply
text). This allows the milter to just set a list of reply codes
without any custom text, which can make writing the milter code a
bit simpler (no need to allocate strings etc). Having both reply
code and reply text of course causes a consistency problem ("does
the code matches the text?"), but that should be already handled
for the existing API.

footnotes:
[1] to make things even simpler for a milter, rtexts could be NULL
too; it's not really required to created an array of NULLs.

[2] it might have been a better API to include an optional reply
text parameter for each xxfi_*() function.

Wietse Venema

2011-11-25 18:54:29 UTC

Post by Wietse Venema
pkt-length command nreplies rcodes*nreplies null-terminated-rtext*nreplies

! int
! smfi_setreplies(SMFICTX *ctx, unsigned int nreplies, int *rcodes, const char **rtexts)
! rcodes: a status array of size nreplies, each element must be one of
! SMFIS_TEMPFAIL, SMFIS_REJECT, SMFIS_ACCEPT
! rtexts: a text array of size nreplies, each element must be either
! NULL or a valid SMTP reply text which matches the corresponding rcode,
[1]

I'm sure libmilter API users will appreciate that the API for PDPR
replies is consistent with that of smfi_setreply / smfi_setmlreply.

I care primarily that the on-the-wire format is consistent with
SMFIR_REPLY, which transits one reply (including code, enhanced
status, text) as one string whether the reply is multiline or not,
and with the same special handling of % characters.

We're simply receiving N replies instead of just one; there is no
need to introduce new formatting.

By the way, sending the counter is redundant. We could drop it as
the Sendmail Milter protocol does not send a counter ahead of an
array of name/value macros, either.

Thus, I simply the proposed format to:

pkt-length SMFIR_REPLIES null-terminated-text+

with one or more null-terminated-text instances, formatted as with
SMFIR_REPLY. The receiver will figure out how many replies there
are, and it can take appropriate action if the number is wrong, or
if the last reply was truncated.

Wietse

Claus Assmann

2011-12-03 01:55:54 UTC

On Fri, Nov 25, 2011, Wietse Venema wrote:

[sorry for the late reply, I was hoping to get more answers from
the people I contacted.]

Post by Wietse Venema
I'm sure libmilter API users will appreciate that the API for PDPR
replies is consistent with that of smfi_setreply / smfi_setmlreply.

It seems I chose a bad name for the function, so let's call it
smfi_prdr() to avoid this kind of confusion. The APIs of smfi_setreply()
and smfi_setmlreply() versus smfi_prdr() are very different:
- smfi_set*reply() are complementary functions that set a text
reply for a return value.
- There is no function that sets the return value for PRDR, smfi_prdr()
does that. If I wanted to "mimic" smfi_set*reply() then I would
have smfi_prdr() (well, actually xxfi_eom() but I can't change that
API) setting return codes and an additional smfi_setreplies() setting
corresponding return texts. I want to avoid this (I consider this
a design failure, but we can't change that easily -- however, we
don't need to repeat it).

The formats of the text replies for smfi_set*reply() and smfi_prdr()
are very different. For smfi_setmlreply() the library "assembles"
the actual reply string based on the components, smfi_prdr() requires
the user the provide the full text -- that makes the library simpler
and it also makes the API simpler.

I contacted some people who worked on milters etc but got only one
reply which agreed with the suggested API, i.e., provide the array
of reply codes and make the text optional (smfi_set*reply() is
seemingly not used very often).

Post by Wietse Venema
I care primarily that the on-the-wire format is consistent with
SMFIR_REPLY, which transits one reply (including code, enhanced

I understand that's an additional complexity, but this seems to
be minor.

Post by Wietse Venema
status, text) as one string whether the reply is multiline or not,
and with the same special handling of % characters.

The handling of '%' seems to be bogus and I don't recall why we did
that. It seems like a mistake (I might take a look at the cvs history
when I have access to it again) -- I don't see how it is useful for
anything. Unless there is a technical argument in its favour, I
prefer to treat the strings as constant (not subject to any
interpretation/modification).

[I might say that I learned a lot from writing libmilter and
developing the protocol; I tried to make it better twice: first in
milter v2 which allows for option negotiation, and then a new version
for MeTA1 without the need for backward compatibility. Hence the
API for smfi_prdr() is different and IMHO better.]

Wietse Venema

2011-12-03 02:12:20 UTC

Post by Claus Assmann
[sorry for the late reply, I was hoping to get more answers from
the people I contacted.]

No problem.

Will the wire protocol be:

int32-pktlen SMFIR_PRDR int32-reply-count reply-text*

Where each reply text is null-terminated, and is of the form

smtp-code SP enhanced-status-code SP text

Where smtp-code is required, the other fields may be absent, and
"text" is a single line without CRLF and without special meaning
for the % character (or any character besides null).

Once we have a wire format I can write some code.

Wietse

Claus Assmann

2011-12-03 02:33:01 UTC

Post by Wietse Venema
int32-pktlen SMFIR_PRDR int32-reply-count reply-text*

It will be your original suggestion:

int32-pkt-length SMFIR_PRDR nreplies int32-rcodes*nreplies rtext*nreplies
where
nreplies = int32-reply-count

Post by Wietse Venema
Where each reply text is null-terminated, and is of the form
smtp-code SP enhanced-status-code SP text
Where smtp-code is required, the other fields may be absent, and
"text" is a single line without CRLF and without special meaning
for the % character (or any character besides null).

Each reply text must be formatted according to the RFCs,
i.e., including CRLF and it can be multi-line:

smtp-code SP enhanced-status-code SP text \r\n

or

smtp-code - enhanced-status-code SP text1 \r\n
smtp-code - enhanced-status-code SP text2 \r\n
...
smtp-code SP enhanced-status-code SP textN \r\n

That is, the text can be presented "as is" to the client. Hence
the MTA doesn't have to do anything extra (of course it is probably
a good idea to check the syntax for correctness -- as the MTA may do
for replies from an SMTP server).

Wietse Venema

2011-12-03 14:51:24 UTC

Post by Wietse Venema
int32-pktlen SMFIR_PRDR int32-reply-count reply-text*

That original suggestion was a mistake, as I pointed out later.

Post by Claus Assmann
int32-pkt-length SMFIR_PRDR nreplies int32-rcodes*nreplies rtext*nreplies
where
nreplies = int32-reply-count

This is inconsistent with the SMFIR_REPLY command. That command
sends each (code, status, text) as one string instead of sending
numeric codes separately. Inconsistency is one of the causes of
programmer error and security problems.

I propose instead:

int32-pktlen SMFIR_PRDR int32-reply-count reply-text*

(BTW The int32-reply-count is redundant, it just is there to make
the protocol more robust).

Post by Claus Assmann
Each reply text must be formatted according to the RFCs,
smtp-code SP enhanced-status-code SP text \r\n

This is inconsistent with SMFIR_REPLY, which does not terminate the
reply with \r\n. Inconsistency like this just introduces new
opportunities for programmer error and security holes.

I propose instead:

smtp-code SP enhanced-status-code SP text NULL

for single-line responses, and

smtp-code '-' enhanced-status-code SP text \r\n
smtp-code SP enhanced-status-code SP text NULL

for multi-line replies.

Sorry for being such a bitch, but I must insist that we don't
reinvent wheels and stick with the existing wheels that already
work in SMFIR_REPLY (I'm fine with dropping the '%' special handling).

Wietse

Claus Assmann

2011-11-26 17:58:11 UTC

Here's a problem: there can be multiple milters (in contrast to
MeTA1 which leaves multiplexing of milters to a different program,
hence simplifying the MTA code):

each milter can add/reject/delete RCPTs and only the last milter
"knows" the actual lists of RCPTs. This will be a mess wrt processing
PRDR in the MTA: it has to keep track of what each milter "believes"
is the list of RCPTs (which is complicated by adding/deleting RCPTs
in xxfi_eom()).

"Simple" solution: only the last milter is allowed to use PRDR (the
MTA could send SMFIP_PRDR only to the last milter to facilitate that).

I'll contact some other people to get their input on PRDR in libmilter
too.

Wietse Venema

2011-11-26 23:35:40 UTC

Post by Claus Assmann
Here's a problem: there can be multiple milters (in contrast to
MeTA1 which leaves multiplexing of milters to a different program,
each milter can add/reject/delete RCPTs and only the last milter
"knows" the actual lists of RCPTs. This will be a mess wrt processing
PRDR in the MTA: it has to keep track of what each milter "believes"
is the list of RCPTs (which is complicated by adding/deleting RCPTs
in xxfi_eom()).

This information can be represented as a list of

(recipient, last Milter that accepted the recipient)

For example, (***@example.com, 2) means that the first two Milters
may send per-recipient end-of-data replies for this recipient, and
that later Milters may not.

Wietse

Wietse Venema

2011-11-26 23:48:31 UTC