SPFv3 proposal: rawfail result

[Michael Deutschmann]

(note: this is somewhat of a re-hash of my earlier "fm="
forwarder-mitigation modifier. But the DKIM connection has been dropped,
and adding this in a new version of SPF would both be more productive and
allow simpler syntax.)

I propose that in SPFv3, we add a new result, called "rawfail" and
selected with the character "/". I call it "rawfail", not "hardfail",
because the difference between it and "fail" is qualitative, not merely a
change in intensity.

"rawfail" recommends that a message be rejected always, ignoring the
possibility that a message may have failed SPF only because it passed
through a traditional forwarder. Coupled with the addition, v3 "fail"
would be explicitly defined to merely recommend rejection *if and only if*
the recipient MX is sure that the message is not a forward.

"fail" remains distinct from "softfail", in that "softfail" does not give
a strong recommendation to reject in the case that the message is known
not to be a a forward.

I don't expect recipient MX's in practice to "obey" rawfail if they in
fact have a forwarder whitelist and the message matches. But that's not
the point. Rather, the goal is to get the many MXes out there that do not
have the information needed to tell forgeries from forwards, to block on
rawfail and not on fail.

While at first glance, getting entities to not block on fail may seem
like a step back, in the long run it is two steps forward. SPF is only as
good as its senderside deployment, and a big problem at present is sites
that publish ?all for no other reason than to make sure their mail gets
through even if it is forwarded. That in turn means that no mailbox --
not even vanity-domain ones where the admin is confident there are no
incoming forwarders -- can be protected from incoming forgeries of those
sites.

Rawfail's main benefit is to remind implementors that fail is not to be
implemented with the semantics we assign to rawfail -- it doesn't need to
be actually used by senders to give this benefit.

Still, it would be a nice option for the sender to have. Some senders
may choose "/all", accepting responsibility for the lower deliverability
in order to supress more forgeries. And it would be of obvious value to
those using the VERP / exists / magic-DNS approach.

It would be especially propitious to do this during an incompatible
upgrade, as that would deny unnecessary-"?all" users the excuse of needing
to get through to sites that haven't upgraded. Such senders could
continue to publish "?all" in their SPFv1 records, and "-all" in their
SPFv3 records. At the same time, this would provide an incentive for
receivers to upgrade.

---- Michael Deutschmann <***@talamasca.ocis.net>

[Stuart D. Gathman]

Post by Michael Deutschmann
"rawfail" recommends that a message be rejected always, ignoring the
possibility that a message may have failed SPF only because it passed
through a traditional forwarder. Coupled with the addition, v3 "fail"
would be explicitly defined to merely recommend rejection *if and only if*
the recipient MX is sure that the message is not a forward.

The forwarding you are concerned about is something only the
recipient can know about or initiate. Why would a sender want a message
rejected if the recipient happened to forward it to another mailbox?

I'm not saying it is stupid - in fact, I'll throw out a possible reason:

A bank wants to send messages directly to recipients via TLS, and
wants recipients to reject emails that came through a forwarder,
and were possibly tampered with.

[Michael Deutschmann]

Post by Stuart D. Gathman
The forwarding you are concerned about is something only the
recipient can know about or initiate. Why would a sender want a message
rejected if the recipient happened to forward it to another mailbox?

Usually, they don't. Except for the case of the VERP/exists/magic-DNS
hack, deploying "/all" would be a reckless act.

But not a senseless one. A (non-VERP) sender who publishes "/all" is not
trying to break forwarding; he would be *accepting* the breakage of
forwarding in return for a much higher efficacy in supressing forgeries.

And again, the key advantage of "/all" is not that many senders will use
it. It's to ensure that recipients don't accidentally assign rawfail
semantics to "-all", a problem that has ruined SPFv1 by deterring senders
from publishing it.

Giving senders the ability to use "/all" to tell a normally-cautious
validator to go wild, is just a bonus.

---- Michael Deutschmann <***@talamasca.ocis.net>

[Stuart D. Gathman]

Post by Michael Deutschmann

Post by Stuart D. Gathman
The forwarding you are concerned about is something only the
recipient can know about or initiate. Why would a sender want a message
rejected if the recipient happened to forward it to another mailbox?

But not a senseless one. A (non-VERP) sender who publishes "/all" is not
trying to break forwarding; he would be *accepting* the breakage of
forwarding in return for a much higher efficacy in supressing forgeries.

So to paraphrase the semantics of "rawfail": even if a receiver does
not track their forwarders (a large legacy ESP, for example), rawfail
asks them to reject a message anyway.

Post by Michael Deutschmann
And again, the key advantage of "/all" is not that many senders will use
it. It's to ensure that recipients don't accidentally assign rawfail
semantics to "-all", a problem that has ruined SPFv1 by deterring senders
from publishing it.

That is a bogus argument. No matter what you do, there will be receivers
that don't actually read the standard. "Rawfail" will not help with that.
No one should avoid publishing "-all" because there are clueless receivers.
It is only by missing important email that they will get off their duff
and figure out what they screwed up (likely one of the common mistakes
listed on openspf.org). Furthermore, the sender does get the reject, and
knows to take action. (Receivers that throw SPF fails or any other reject
into the bit bucket are a hopeless case.)

I do see potential usefulness in requesting that forwarded messages get
rejected. It could help ensure a direct transfer between sender and receiver,
reducing the likelihood of tampering (especially in conjuction with TLS
between the MTAs). Although signed S/MIME is *much* more reliable, typical
end users couldn't generate a key pair to save their life.

[Michael Deutschmann]

Post by Stuart D. Gathman
So to paraphrase the semantics of "rawfail": even if a receiver does
not track their forwarders (a large legacy ESP, for example), rawfail
asks them to reject a message anyway.

Exactly.

Although I object to the namecalling -- unless and until a "TENBOX"
standard emerges, it is unfair to call the ESPs in question "legacy".
Today, forwarder whitelisting requires ad hoc approaches and is generally
only available when the end user and the mailserver admin are the same
person.

Post by Stuart D. Gathman

Post by Michael Deutschmann
And again, the key advantage of "/all" is not that many senders will use
it. It's to ensure that recipients don't accidentally assign rawfail
semantics to "-all", a problem that has ruined SPFv1 by deterring senders
from publishing it.

That is a bogus argument. No matter what you do, there will be receivers
that don't actually read the standard. "Rawfail" will not help with that.

But it would improve things, as even in this very forum there is not
universal agreement that SPFv1 "-all" is not a raw fail.

The root problem is that the original designers of SPFv1 arrogantly
assumed that SRS deployment would quickly outpace receiverside SPFv1
deployment, hence there would be no need to make the distinction.

Post by Stuart D. Gathman
No one should avoid publishing "-all" because there are clueless receivers.

But they do. That annoys me, but we cannot force them to stop lying
(saying "?all" when the truth is "-all"). All we can do is reduce the
temptation by cutting down the number of "clueless receivers".

Post by Stuart D. Gathman
I do see potential usefulness in requesting that forwarded messages get
rejected. It could help ensure a direct transfer between sender and receiver,

"/all" is insufficent for that purpose, as it will not block SRS
forwarders, or pull-based arrangements.

---- Michael Deutschmann <***@talamasca.ocis.net>

[Stuart D Gathman]

Post by Michael Deutschmann

Post by Stuart D. Gathman
I do see potential usefulness in requesting that forwarded messages get
rejected. It could help ensure a direct transfer between sender and receiver,

"/all" is insufficent for that purpose, as it will not block SRS
forwarders, or pull-based arrangements.

SRS documents the forward - your domain is not in the return-path (apart
from being encoded in the localpart). Pull-based is still direct.

[Alessandro Vesely]

Post by Michael Deutschmann

No matter what you do, there will be receivers that don't actually
read the standard.

The standard says

A "Fail" result is an explicit statement that the client is not
authorized to use the domain in the given identity. The checking
software can choose to mark the mail based on this or to reject the
mail outright.

Post by Michael Deutschmann
But it would improve things, as even in this very forum there is not
universal agreement that SPFv1 "-all" is not a raw fail.
The root problem is that the original designers of SPFv1 arrogantly
assumed that SRS deployment would quickly outpace receiverside SPFv1
deployment, hence there would be no need to make the distinction.

IME, services that mailout from web pages and forwarders have learned
to set an empty mailfrom in case nobody is interested in knowing about
possible failures, or an SPF-compatible address otherwise.

Post by Michael Deutschmann

No one should avoid publishing "-all" because there are clueless receivers.

How about clueless forwarders?

Post by Michael Deutschmann
But they do. That annoys me, but we cannot force them to stop lying
(saying "?all" when the truth is "-all").

"Truth"? Do you mean whether it is true that a domain wants clueless
senders to get blocked rather than marked?

IMHO there is enough confusion already with neutral and softfail. If
we want to provide for more, and still not block, why don't we just
allow to set the "mark" value numerically, specifying the score that
should be added or subtracted? E.g.

"v=spf3 +(5)62.94.243.226 -(10)all" ; unluckily signs are reversed

Post by Michael Deutschmann

I do see potential usefulness in requesting that forwarded messages get
rejected. It could help ensure a direct transfer between sender and receiver,

"/all" is insufficent for that purpose, as it will not block SRS
forwarders, or pull-based arrangements.

[Michael Deutschmann]

Post by Alessandro Vesely
IMHO there is enough confusion already with neutral and softfail. If
we want to provide for more, and still not block, why don't we just
allow to set the "mark" value numerically, specifying the score that
should be added or subtracted? E.g.

The distinction between rawfail and fail is not quantitative -- this is
why I call it "rawfail" and not "hardfail".

Fail still means 100.00% confidence of rejection *if the validator is sure
the message is not a forward*.

Rawfail makes the *qualitative* change of telling the validator not to
worry about the risk of forwarding false positives.


If 5 outcomes (pass,neutral,softfail,fail,rawfail) is really too much,
then I nominate softfail for removal in SPFv3 to make room for rawfail.

---- Michael Deutschmann <***@talamasca.ocis.net>

[Alessandro Vesely]

Post by Michael Deutschmann

Post by Alessandro Vesely
IMHO there is enough confusion already with neutral and softfail. If
we want to provide for more, and still not block, why don't we just
allow to set the "mark" value numerically, specifying the score that
should be added or subtracted? E.g.

The distinction between rawfail and fail is not quantitative -- this is
why I call it "rawfail" and not "hardfail".
Fail still means 100.00% confidence of rejection *if the validator is sure
the message is not a forward*.

To me, it seems nearly impossible to formally define that difference
within the definition of the check_host function, in the face of
forgeries.

[Michael Deutschmann]

Post by Alessandro Vesely
To me, it seems nearly impossible to formally define that difference
within the definition of the check_host function, in the face of
forgeries.

Actually, "check_host" doesn't need to worry about it. It would be
called upon only to *return* "fail" or "rawfail" according to the SPF
records, not to *interpret* them.

Still, I see what you're talking about, but would counter that it is even
harder to write a single reference behavior-set that makes use of the
distinction between "unknown" and "pass", let alone "softfail".

What I'd do is define a second function "is_internal_forward", that
returns a ternary result : True, False, or Unknown. The definition would
have to be supplied by the local site, but it would have to satisfy the
following requirements.

* MUST return True in all cases where a backup MX is handing off a
message towards the primary
* MAY return Unknown in any other case
* SHOULD NOT return True when the message is not a forward.
* SHOULD NOT return False when the message is a forward.

(I use SHOULD because today forwarder whitelisting is done in an ad-hoc
way that could be utterly broken by a forwarder changing the IP or
hostname of its handoff mailserver without notice.

If we used MUSTs, then reject-on-ordinary-fail would not be available to
sites that have merely enumerated their incoming non-rewriting forwards,
but only to those confident that there are *none at all*)

Then the SPF reject vote is decided by the following matrix:

rawfail fail softfail unknown pass
True Accept Accept Accept Accept Accept
Unknown Reject Accept Accept Accept Accept
False Reject Reject Accept Accept Accept

---- Michael Deutschmann <***@talamasca.ocis.net>

[Stuart D. Gathman]

Post by Michael Deutschmann
(I use SHOULD because today forwarder whitelisting is done in an ad-hoc
way that could be utterly broken by a forwarder changing the IP or
hostname of its handoff mailserver without notice.

The whitelisting is robust for most alias forwarders when you whitelist the
forwarder domain, and use SPF or best guess to identify mail from the
forwarder. When attempting the explain this before, I called it
a "pretend" MAIL FROM. The algorithm in its simple form goes like this:

if pending reject due to SPF fail or softfail:
for all whitelisted alias forwarder domains:
check SPF with connect IP and forwarder domain as MAIL FROM
if pass:
assume mail was from alias forwarder

The two difficulties with this are:

1) It is not always obvious what the domain of an alias forwarder is.
It could be the original RCPT domain in the case of a university, or
a separate business domain in the case of a commercial alias forwarder,
or something else entirely. It is the domain they would use in MAIL FROM
if they were to use SRS. GRIPE: If forwarders aren't going to implement
SRS, they could at least let customers know their domain...

2) Doing additional SPF checks for more than 2 or 3 forwarder domains
is expensive. This can be optimized by "compiling" the list into an
IP set ala libspf2 - but this increases the complexity of the implementation.

[Michael Deutschmann]

Post by Stuart D. Gathman
The whitelisting is robust for most alias forwarders when you whitelist the
forwarder domain, and use SPF or best guess to identify mail from the
forwarder. When attempting the explain this before, I called it

That doesn't help with the scenario that worries me:

A user subscribes to ISP A for some time, but then switches to ISP B.
ISP A graciously sets up a forward from his old mailbox to his new one at
ISP B (this is also the situation where the forwarder is least motivated
to try something like SRS).

ISP B and the user are quite hip mail-security wise, so they arrange a
whitelisting for the servers ISP A has been using for the handoff, and
then enable reject-on-SPF-fail for other cases.

Then ISP C buys out ISP A, and consolidates mail handling at one data
center. The forwards now come from a different IP, different rDNS domain,
and different HELO. The whitelisting breaks.

(And then users at ISP D notice they can't mail to the forwarded mailbox,
and lobby ISP D to use "?all" instead of "-all" to work around the
problem...)

The only cure for this problem is for ISP A and ISP B to agree on how the
whitelisting is to be done. Then ISP A will ensure that whatever indicia
are being used remain stable, or at the very least warn ISP B before
changing the handoff server.

Aside from the above scenario, merely temporarily disabling SPF to let a
few forwards through, observing the IP address, hostname and HELO, and
then whitelisting whatever seems most stable should work well enough.

---- Michael Deutschmann <***@talamasca.ocis.net>

[Stuart D. Gathman]

Post by Michael Deutschmann

Post by Stuart D. Gathman
The whitelisting is robust for most alias forwarders when you whitelist the
forwarder domain, and use SPF or best guess to identify mail from the
forwarder. When attempting the explain this before, I called it

A user subscribes to ISP A for some time, but then switches to ISP B.
ISP A graciously sets up a forward from his old mailbox to his new one at
ISP B (this is also the situation where the forwarder is least motivated
to try something like SRS).
ISP B and the user are quite hip mail-security wise, so they arrange a
whitelisting for the servers ISP A has been using for the handoff, and
then enable reject-on-SPF-fail for other cases.

You missed the part where you whitelist the DOMAIN, not server IPs. Using
SPF or best guess if that is not available.

Post by Michael Deutschmann
Then ISP C buys out ISP A, and consolidates mail handling at one data
center. The forwards now come from a different IP, different rDNS domain,
and different HELO. The whitelisting breaks.

Doesn't break if the SPF record is properly transitioned. You're right
though that "best guess" would probably break. That is why we promote SPF.

[Michael Deutschmann]

Post by Stuart D. Gathman
Doesn't break if the SPF record is properly transitioned. You're right
though that "best guess" would probably break. That is why we promote SPF.

Your hack may still break, because supporting it is not currently a
documented consideration in deciding what to publish.

Say ISP C decides to phase out all use of the ISP A domain, except to
forward mail from the legacy addresses to each customer's new address at
ISP C's domain. Then, since no further legitimate mail is expected to
bear the ISP A domain in the envelope sender, "v=spf1 -all" becomes a
reasonable SPF record for that domain.

---- Michael Deutschmann <***@talamasca.ocis.net>

[Stuart D. Gathman]

Post by Michael Deutschmann

Post by Stuart D. Gathman
Doesn't break if the SPF record is properly transitioned. You're right
though that "best guess" would probably break. That is why we promote SPF.

Your hack may still break, because supporting it is not currently a
documented consideration in deciding what to publish.

Then maybe we should make it one. It is a simpler and easier alternative to
deploying SRS (although more trouble for the final receiver).

[Michael Deutschmann]

Post by Stuart D. Gathman
Then maybe we should make it one. It is a simpler and easier alternative to
deploying SRS (although more trouble for the final receiver).

It would work better to promulgate Sham SRS, which would avoid the
scalability problem of your hack.

(To those not paying attention to the forums in 2008, Sham SRS means
discarding the SRS goal of transmitting bounce messages back to the
original sender. Instead, the envelope sender is rewritten to a constant
value, which accepts no mail.)


But this is a distraction from the question I originally pondered, which
is -- given a site that has whitelisted all its friendly incoming
forwards, yet is using a whitelisting heuristic that the forwarding
entities have not promised not to break, can it reject on an ordinary fail
when the whitelist engine says "not a trusted forward"?

---- Michael Deutschmann <***@talamasca.ocis.net>

[Stuart D. Gathman]

Post by Michael Deutschmann

Post by Stuart D. Gathman
Then maybe we should make it one. It is a simpler and easier alternative to
deploying SRS (although more trouble for the final receiver).

It would work better to promulgate Sham SRS, which would avoid the
scalability problem of your hack.

As a less clueless than average email sender, I would much prefer
having the email simply rejected by SPF fail to Sham SRS. The DSN
from the forwarder would contain the new email, and I would simply resend to
that. Sham SRS gives no notice of delivery problems (unless the forward
is the final hop). I realize the average email user never actually reads DSNs,
so this is not a general solution.

What if there was a standard SMTP code for SPF fail? Then a non-SRS forwarder
could easily send an end-user friendly DSN to facilitate resending
to the new email?

Post by Michael Deutschmann
But this is a distraction from the question I originally pondered, which
is -- given a site that has whitelisted all its friendly incoming
forwards, yet is using a whitelisting heuristic that the forwarding
entities have not promised not to break, can it reject on an ordinary fail
when the whitelist engine says "not a trusted forward"?

I say "yes". But I also realize that this is a judgement that
others may disagree with (not a black and white issue). Keep in mind
that rejecting on SPF fail after alias forwarding is not such a horrible thing.
If the forwarder MTA is standards compliant, the user is notified of the new
address, and can resend. At least the email doesn't go in the bit bucket.

[Stuart D. Gathman]

Post by Alessandro Vesely

Post by Stuart D. Gathman
No one should avoid publishing "-all" because there are clueless receivers.

How about clueless forwarders?

Receivers choose their (legitimate) forwarders, so a clueless forwarder
is still a clueless receiver. (A receiver with a clue would certainly
*not* want email "forwarded" by random MTAs that they did not request to
do so.) If a receiver is forced to deal with a badly configured forwarder,
then a simple whitelist accomodates that without causing problems with SPF.

[Stuart D. Gathman]

Post by Michael Deutschmann
The root problem is that the original designers of SPFv1 arrogantly
assumed that SRS deployment would quickly outpace receiverside SPFv1
deployment, hence there would be no need to make the distinction.

If it is not clear in the spec that you SHOULD NOT reject on -all
when not actually evaluated on an MX for the original RCPT TO domain, then we
should make that clear.

This *does* make -all not very useful to a receiver when the incoming border
MTAs are not clearly defined (since alias forwarders are the MXs for the
original RCPT TO domain).

I do not support adding "rawfail" simply to clarify the meaning of "fail".
I am open to adding "rawfail" for its own sake, as a request to reject
mail rather than deliver via alias forwarding.

Some of the arguments used to support always rejecting on SPF fail apply to
rawfail: the reject DSN contains the real address, and a savvy sender can
simply resend, possibly updating his address book. So rawfail could be
a useful option for a Sender Policy. (In addition to the already mentioned
feature of requesting direct delivery.)

[Alessandro Vesely]

Post by Stuart D. Gathman
If it is not clear in the spec that you SHOULD NOT reject on -all
when not actually evaluated on an MX for the original RCPT TO domain, then we
should make that clear.

Yes, please. The spec mentions that checking "can be performed
elsewhere in the mail processing chain", but then it says (twice)

<ip> - the IP address of the SMTP client that is emitting the
mail, either IPv4 or IPv6.

Formally, the client that /is emitting/ the mail is the previous hop,
isn't it? IMHO, that deserves an errata.

Perhaps the correct wording could be based on a formal definition of
border, along the lines of http://www.openspf.org/Community/Glossary