About redirects

[Andre Nathan]

Hello

Imagine the following situation:

***@example.com sends an email to ***@local1.com and ***@local2.com. Both
@local*.com accounts, via virtual_alias_maps, redirect to ***@external.com
and ***@external.com.

Am I right to assume that Postfix will create a single SMTP session with
the SMTP server at external.com? That is, a single message with MAIL
FROM = ***@example.com and RCPT TO = ***@external.com and RCPT TO =
***@external.com will be sent?

If that is indeed the case, is there a configuration directive that
forces two separate messages to be sent? The intent here is to be able
to use the correct forward domain when SRS-signing the messages, ie, use
local1.com as the SRS domain in the first message and local2.com in the
second.

Thanks in advance,
Andre

[Michael Tokarev]

Post by Andre Nathan
Hello
@local*.com accounts, via virtual_alias_maps, redirect to ***@external.com

Do you have a patch or an implementation comments about some feature
you want to implement?

Thanks,

/mjt

[Andre Nathan]

Hi Michael

Post by Michael Tokarev
Do you have a patch or an implementation comments about some feature
you want to implement?

I am implementing an SRS-enabled SPF milter to work with postfix and was
wondering how to deal with this case.

Thanks,
Andre

[Rod Dorman]

Post by Andre Nathan

Post by Michael Tokarev
Do you have a patch or an implementation comments about some feature
you want to implement?

I am implementing an SRS-enabled SPF milter to work with postfix and was
wondering how to deal with this case.

I believe he was subtly pointing out that this list is for Postfix
development questions, not for Postfix configuration questions.

[Wietse Venema]

Post by Andre Nathan
Hello

Postfix by default delivers different domains in different SMTP
sessions.
If this aliasing is done before Postfix delivers the mail, then the
domains are no longer different. Postfix will therefore deliver
both in the same SMTP session.

Post by Andre Nathan
Am I right to assume that Postfix will create a single SMTP session with
the SMTP server at external.com? That is, a single message with MAIL

Of course. One message has two recipients in the same domain.

Post by Andre Nathan
If that is indeed the case, is there a configuration directive that
forces two separate messages to be sent? The intent here is to be able
to use the correct forward domain when SRS-signing the messages, ie, use
local1.com as the SRS domain in the first message and local2.com in the
second.

You could dumb down Postfix and reduce the default_recipient_limit.
I don't know if is a good idea to make the queue scheduler dependent
on both the final and the initial recipient address.

Wietse

[Andre Nathan]

Post by Wietse Venema
You could dumb down Postfix and reduce the default_recipient_limit.
I don't know if is a good idea to make the queue scheduler dependent
on both the final and the initial recipient address.

Is there anything a milter application could do to force delivery using
two separate SMTP sessions?

Thanks,
Andre

[Wietse Venema]

This post might be inappropriate. Click to display it.

[Andre Nathan]

Hi Wietse

Post by Wietse Venema
In your case the scheduler could compare the orig_rcpt and rcpt and
if they differ, switch to single-recipient mode. However, I don't
want to maintain hard-coded versions of DKIM or SRS etcetera inside
Postfix. That is why I provided the Milter and SMTP policy extension
interfaces. In the case of SRS this would require an additional
extension interface for the scheduler.

I have to clarify an aspect about SRS: it does not always require
single-recipient deliveries as my example seemed to have led people here
to think it does. To see it, extend the previous example so that the
message sent from ***@example.com is now sent, as before, to ***@local1.com
and ***@local2.com, and also to ***@local2.com, which redirects to
***@external.com. Now the redirects would result in a message from
***@example.com to ***@external.com, a message from ***@example.com to
***@external.com and a third message from ***@example.com to ***@external.com.
The first one would use local1.com as the SRS domain, while the other
two would use local2.com. These two latter messages could share the same
SMTP session for delivery.

With the previous clarification in mind, the scheduler API seems the
cleanest path to follow because:

1. As you pointed before, splitting messages in the queue would increase
file system load. Adding this price tag to SRS would lead mail provider
to think twice before supporting SRS.

2. As explained before, SRS doesn't always require single-recipient
deliveries, so one doesn't have to waste bandwidth when it's possible to
group deliveries; more one this subject when I present my suggestion for
the scheduler API below.

3. If such scheme is implemented, VERP could then be removed from
Postfix's core, and be implemented as a special case of what was
described above, that is, the messages wouldn't be grouped for delivery.

4. Implementing the ability to modify the envelope sender via the
scheduler API would also create an additional advantage: no more
virtual_alias_maps lookups than the ones already done by Postfix would
be necessary. Currently, a milter application needs a way to resolve
redirects by itself, because it doesn’t have access to the results of
Postfix’s own resolution (as previously discussed in
http://thread.gmane.org/gmane.mail.postfix.devel/2557).

If a scheduler API is to be implemented in Postfix, here is my
suggestion of what it should accomplish:

1. Ideally, one should be able to not only tell the scheduler to switch
to single-recipient delivery mode, which, as described above, would
result in some inefficiency, but how to group recipients according to
some criteria (defined by the API user).

2. The scheduler API must allow actions to be taken when the message is
being sent. In the case of SRS specifically, this action would be to
change the envelope sender address.

3. SRS can't decide how to group the recipients just comparing orig_rcpt
and rcpt, for example, if all local recipients are delivered using LMTP,
orig_rcpt may always be different to rcpt. Moreover, if a message is
added to the queue by a SASL-authenticated SMTP session, the messaging
isn't being redirected, but being sent out, thus SRS is not even
necessary. These cases imply that the scheduler API must provide access
to the metadata associated to the message in the queue to allow API
users to make decisions on how to group the recipients.

Once this new API is available, SPF would be better implemented using
the policy API since what it's really doing is enforcing policy, and SRS
using the scheduler API to be efficient. The milter API wouldn't be
necessary for SPF/SRS anymore.

Andre

[Wietse Venema]

Andre Nathan:
[ Charset UTF-8 unsupported, converting... ]

Post by Andre Nathan
Hi Wietse

Post by Wietse Venema
In your case the scheduler could compare the orig_rcpt and rcpt and
if they differ, switch to single-recipient mode. However, I don't
want to maintain hard-coded versions of DKIM or SRS etcetera inside
Postfix. That is why I provided the Milter and SMTP policy extension
interfaces. In the case of SRS this would require an additional
extension interface for the scheduler.

I have to clarify an aspect about SRS: it does not always require
single-recipient deliveries as my example seemed to have led people here
to think it does. To see it, extend the previous example so that the

Sure, but sending recipients one-at-a-time would be safe.

The parts of the scheduler that deliver mail are something that I
would not tinker with. This is an intricate machine.

Perhaps a safe approach is to move all code out of the scheduler
that reads receipients from file, and that invokes the trivial-rewrite
daemon to figure out where to deliver. The result of this "preprocessing"
stage is a grouping of recipients into clusters that can be delivered
in the same mail delivery transaction.

With VERP. the clusters would always have size 1, and with SRS, the
clusters may have size >1. In both cases each cluster of recipients
has its own envelope sender.

The scheduler API (actually a protocol) would then receive requests
containing (sender, recipients) and it would reply with one or more
(sender, transport, nexthop, recipients) responses. The scheduler
would append these to its per-transport-per-destination queues.

Wietse

[Andre Nathan]

Post by Wietse Venema
The scheduler API (actually a protocol) would then receive requests
containing (sender, recipients) and it would reply with one or more
(sender, transport, nexthop, recipients) responses. The scheduler
would append these to its per-transport-per-destination queues.

This would be great! I think the only thing missing in the reply is a
field indicating whether the sender is SASL-authenticated, so that one
can differentiate between an outgoing message sent by a local user and a
redirect.

Andre

[Wietse Venema]

Post by Andre Nathan

Post by Wietse Venema
The scheduler API (actually a protocol) would then receive requests
containing (sender, recipients) and it would reply with one or more
(sender, transport, nexthop, recipients) responses. The scheduler
would append these to its per-transport-per-destination queues.

This would be great! I think the only thing missing in the reply is a
field indicating whether the sender is SASL-authenticated, so that one
can differentiate between an outgoing message sent by a local user and a
redirect.

That would be a mistake.

First, an MTA can require SASL authentication for email that isn't
an original submision. Second, an MTA can authenticate clients via
means other than SASL. Third, mail submitted with the Postfix
sendmail command can be an original submission, or it can be a relay
submission after a content filter. An SRS tool should handle these
scenarios correctly or else there will be problems.

Finally, Postfix does not have a "this is forwarded/relayed mail"
field in the queue file, because nothing inside Postfix cares about
such things. Only the SMTP daemon has a notion of "final destination"
that everyone can send mail to, while all other destinations require
some form of additional client authorization.

If I am not mistaken, a sender needs to be SRS-munged when both the
sender and the final recipient are remote for some definition of
"remote". Also, if the sender address is not "remote" then it is a
good idea to have evidence in the form of authentication.

Wietse

[Andre Nathan]

Post by Wietse Venema
That would be a mistake.

Understood. You are correct, of course.

Would you consider then the possibility of adding a tag to the message
metadata in the queue?

The idea is that different smtpd instances could tag the messages they
add to the queue with some user-configurable string, which would then be
exposed by the scheduler protocol. An SRS application could then inspect
the tag value and use it as one of the factors to decide if it will
munge the sender address or not.

The motivation behind the tag is to avoid the need for a lookup on the
sender address (which could even be in a different data store than the
redirect database).

Thanks,
Andre

[Wietse Venema]

Post by Andre Nathan

Post by Wietse Venema
That would be a mistake.

Understood. You are correct, of course.
Would you consider then the possibility of adding a tag to the message
metadata in the queue?
The idea is that different smtpd instances could tag the messages they
add to the queue with some user-configurable string, which would then be
exposed by the scheduler protocol. An SRS application could then inspect
the tag value and use it as one of the factors to decide if it will
munge the sender address or not.
The motivation behind the tag is to avoid the need for a lookup on the
sender address (which could even be in a different data store than the
redirect database).

How is this supposed to work with external content filters, such
as before-queue filters (smtpd_proxy) or post-queue filters
(amavisd-new over SMTP, and pipe(8)->filter->sendmail(1) over UNIX
pipes)?

Wietse

[Andre Nathan]

Post by Wietse Venema
How is this supposed to work with external content filters, such
as before-queue filters (smtpd_proxy) or post-queue filters
(amavisd-new over SMTP, and pipe(8)->filter->sendmail(1) over UNIX
pipes)?

The tag would always be added by the program that injects the message
into the queue.

In the example above, the last step in the chain would add the tag,
which means the smtpd instance that receives the message from the
content filter would set the tag.

Andre

[Wietse Venema]

Post by Andre Nathan

Post by Wietse Venema
How is this supposed to work with external content filters, such
as before-queue filters (smtpd_proxy) or post-queue filters
(amavisd-new over SMTP, and pipe(8)->filter->sendmail(1) over UNIX
pipes)?

The tag would always be added by the program that injects the message
into the queue.
In the example above, the last step in the chain would add the tag,
which means the smtpd instance that receives the message from the
content filter would set the tag.

Many sites have external content filters as described above.

1 - Postfix receives mail from the network with the Postfix SMTP
server. This may be an original submission or not. Or, Postfix
receives mail with the Postfix sendmail command. This may also
be an original submission or not.

2 - Postfix sends the mail through the filter via the Postfix SMTP
client or via the pipe(8) delivery agent. The filter sees the
original recipients; virtual aliasing happens after the filter.

3 - Postfix receives a perhaps modified message from the filter via
the Postfix SMTP server or via the Postfix sendmail command.

4 - Postfix does some virtual alias expansions on recipients, and
then it delivers the message to local or remote recipients.

How should SRS work in this configuration?

Wietse

[Viktor Dukhovni]

Post by Wietse Venema
How should SRS work in this configuration?

If SRS is to be used at all, it needs to be used in the "final"
(after all filters, ...) output stage of an otherwise "inbound"
MTA. In this "final" output stage, the default_transport (outbound
mail) would be configured with a fixed nexthop that does the SRS
processing (one recipient at a time).

A sender-dependendent default transport table would exclude local
sender addresses (though best if these never reach this "inbound"
MTA in the first place) from the SRS destination and deliver these
directly.

With this, no architecture changes required. The OP's mistake is
to treat SRS as in input transformation (Postfix typically does
filtering on input), but in reality SRS is an output transformation
and must thus be delayed until a message is ready to leave.

[Andre Nathan]

Hi Wietse

Post by Wietse Venema
Many sites have external content filters as described above.
1 - Postfix receives mail from the network with the Postfix SMTP
server. This may be an original submission or not. Or, Postfix
receives mail with the Postfix sendmail command. This may also
be an original submission or not.
2 - Postfix sends the mail through the filter via the Postfix SMTP
client or via the pipe(8) delivery agent. The filter sees the
original recipients; virtual aliasing happens after the filter.
3 - Postfix receives a perhaps modified message from the filter via
the Postfix SMTP server or via the Postfix sendmail command.
4 - Postfix does some virtual alias expansions on recipients, and
then it delivers the message to local or remote recipients.
How should SRS work in this configuration?

SRS will work if the proposed scheduler protocol is run on the 4th step
you described. At this point, the SRS application will make a request
"(sender, recipients)" and receive "(sender, transport, nexthop,
translated_recipients, tag)" replies. The recipients would be grouped
into clusters, and translated_recipients would be inspected to decide if
the recipients are local or not. The tag would be inspected to decide if
the sender is local or not. If both are remote, then the envelope-from
will be SRS-munged.

In this description, I'm assuming that Postfix was configured in such a
way that meaningful tags can be chosen on the appropriate smtpd
instances. This means that two "mail paths" would have to be configured,
for inbound and outbound mail. Thus, one would have to setup two
different smtpd instances, each connected to their own content_filter
instance, which would in turn send the messages back to Postfix via two
more separate smtpd instances. It is in these two last instantes that
the tag could be applied.

Andre

[Wietse Venema]

Post by Andre Nathan
Hi Wietse

Post by Wietse Venema
Many sites have external content filters as described above.
1 - Postfix receives mail from the network with the Postfix SMTP
server. This may be an original submission or not. Or, Postfix
receives mail with the Postfix sendmail command. This may also
be an original submission or not.
2 - Postfix sends the mail through the filter via the Postfix SMTP
client or via the pipe(8) delivery agent. The filter sees the
original recipients; virtual aliasing happens after the filter.
3 - Postfix receives a perhaps modified message from the filter via
the Postfix SMTP server or via the Postfix sendmail command.
4 - Postfix does some virtual alias expansions on recipients, and
then it delivers the message to local or remote recipients.
How should SRS work in this configuration?

SRS will work if the proposed scheduler protocol is run on the 4th step
you described. At this point, the SRS application will make a request
"(sender, recipients)" and receive "(sender, transport, nexthop,
translated_recipients, tag)" replies. The recipients would be grouped

What tag? What originates it? What brings the tag through the filter?

Did you read and understand Victor's mail? Tags should be unnecessary.
SRS munging is an output function. It does not depend on client
identity.

Wietse

[Andre Nathan]

Post by Wietse Venema
What tag? What originates it? What brings the tag through the filter?
Did you read and understand Victor's mail? Tags should be unnecessary.
SRS munging is an output function. It does not depend on client
identity.

The tag would be created by the system administrator and configured in
master.cf, in the appropriate smtpd instance that receives the message
back from the filter, as is usually done with per-instance
configuration.

I did read Viktor's email, but the ideas here are in the context of the
scheduler protocol being proposed. With the protocol, there would be no
need for one-at-a-time delivery and there's no need to duplicate lookup
resolution. It would also allow the removal of VERP from Postfix's core,
and avoid extra file system load.

Andre

[Stefan Sels]

Post by Andre Nathan

Post by Wietse Venema
What tag? What originates it? What brings the tag through the filter?
Did you read and understand Victor's mail? Tags should be unnecessary.
SRS munging is an output function. It does not depend on client
identity.

The tag would be created by the system administrator and configured in
master.cf, in the appropriate smtpd instance that receives the message
back from the filter, as is usually done with per-instance
configuration.
I did read Viktor's email, but the ideas here are in the context of the
scheduler protocol being proposed. With the protocol, there would be no
need for one-at-a-time delivery and there's no need to duplicate lookup
resolution. It would also allow the removal of VERP from Postfix's core,
and avoid extra file system load.

ok. please stop discussing now.

as pointed out, postfix works in enviroments where more than one
instance (amavisd, etc) is called to process mail on the same host.

There is no way that such tags could be integrated in such setups,
because that protocol would be a way to directly access the sheduler so
it works only for very specific use cases. Amavisd & Co forward via
SMTP, the protocol you are dreaming of - nobody is implementing it -
would require direct API calls outside of SMTP.

Post by Andre Nathan
Andre

greetings,
Stefan

[Viktor Dukhovni]

Post by Andre Nathan

Post by Wietse Venema
You could dumb down Postfix and reduce the default_recipient_limit.
I don't know if is a good idea to make the queue scheduler dependent
on both the final and the initial recipient address.

Is there anything a milter application could do to force delivery using
two separate SMTP sessions?

This makes little sense, milters happen when mail is coming in,
not when it is going out. If a message comes in with multiple
recipients, it stays that way until after the milter processing is
done. The queued message is identical for all recipients.

To handle SRS, you need to intercept the copy of the message that
will be forwarded for a particular recipient. This is only possible
with indirect local(8) delivery, with forwarding via the aliases(5)
database. In this case a new queue-id is created for the single
recipient message that will be sent out after aliasing.

Another approach is to route all forwarded recipients to a second
Postfix instance, via a transport with a recipient limit of 1. The
SRS milter can then be installed there.