RFC 4408 Errata - PermError on invalid domains after macro expansion

[Scott Kitterman]

As I've mentioned previously, I'm working on updating the SPF RFC, RFC
4408, and hoping to get it out of Experimental and onto standards track.

I've gone through the RFC 4408 errata and they all seem quite
straightforward, except for one (copied here, since the web site is down

Be aware that if the domain owner uses macros (Section 8), it is
possible that this result is due to the checked identities having an
unexpected format.
A <target-name> after macro expansion of <domain-spec> can be
invalid, i.e. a string not suited for DNS queries like foo..example
(adjacent dots), a <target-name> with overlong labels, or similar
issues not permitted in the DNS syntax.
The last sentence in the Permerror definition is misleading. A
syntactically malformed <target-name> can be also handled as no
Be aware that if the domain owner uses macros (Section 8), it is
possible that this result is due to the checked identities having an
unexpected format.
Please note that an unexpected <target-name> can be also handled as
no match, ideally implementations document how they handle such
issues. The outcome for an unexpected <domain-spec> without macros
might differ from the outcome for an unexpected <target-name> after
macro expansion.
The last sentence in the Permerror definition is misleading. A
syntactically malformed <target-name> can be also handled as no
Be aware that it is also possible that this result is generated by
certain SPF clients due to the input arguments having an unexpected
format; see Section 4.8.
Note: Historically, this document has made no provisions for how to
handle <domain-spec>s, or macro-expansions thereof, that are
syntactically invalid per [RFC1035], such as names with empty labels
(e.g., "foo..example.com") or overlong labels (more than 63
characters). Some implementations choose to treat as a no-match
mechanisms, and ignore modifiers, with such names, whereas others
throw a "PermError" exception. The outcome for an unexpected
<domain-spec> without macros might even differ from that for an
unexpected <target-name> after macro expansion.
Reporting a TempError in such cases is no option, the syntax problem
won't go away for a given sender policy, HELO identity, envelope
sender address, and sending IP combination with a try again later
TempError approach. If anything else is as expected the sender policy
might need to be fixed manually, supporting PermError.
If the DNS syntax problem is caused by random net abuse, or
intentionally by an attacker, a no match approach, eventually
reaching a FAIL for -all or whatever the sender policy offers, is
better. In common practice this problem is handled as no match OR
PermError, like similar problems explicitly addressed in the
specification.

My preference here would be to treat these as PermError since an empty
domain-spec is not useful to the protocol and should be an error. I
think it's enough of a corner case that I don't think we need to warn
about the difference from RFC 4408. My solution (note: different than

A "PermError" result means that the domain's published records
could not be correctly interpreted. This signals an error
condition that requires manual intervention to be resolved, as
opposed to the TempError result. If the message is rejected
during the SMTP transaction for this reason, the software SHOULD
use an SMTP reply code of 550 and, if supported, the 5.5.2 DSN
code. Be aware that if the domain owner uses macros
(<xref target="macros"/>), it is possible that this result is due
to the checked identities having an unexpected format. The domain
and domain-spec MUST be fully macro expanded before being tested
for errors.

I don't think anything other than macro expansion before evaluation
makes sense. Please discuss.

Scott K

[Commerco WebMaster]

Scott,

I think the second is better. It appears to (rightly) push the issue of
any DNS specific syntactically invalid elements introduced by a
publisher / domain holder within their zone file back to the relevant
DNS related RFC.

That seems more clear and appropriate.

Also you might want to further simplify the wording from
"...certain SPF clients due to the input arguments having..."
to
"...certain SPF clients from input arguments having..."

Best,

Alan M.

Post by Scott Kitterman
As I've mentioned previously, I'm working on updating the SPF RFC, RFC
4408, and hoping to get it out of Experimental and onto standards track.
I've gone through the RFC 4408 errata and they all seem quite
straightforward, except for one (copied here, since the web site is down

Be aware that if the domain owner uses macros (Section 8), it is
possible that this result is due to the checked identities having an
unexpected format.
A <target-name> after macro expansion of <domain-spec> can be
invalid, i.e. a string not suited for DNS queries like foo..example
(adjacent dots), a <target-name> with overlong labels, or similar
issues not permitted in the DNS syntax.
The last sentence in the Permerror definition is misleading. A
syntactically malformed <target-name> can be also handled as no
Be aware that if the domain owner uses macros (Section 8), it is
possible that this result is due to the checked identities having an
unexpected format.
Please note that an unexpected <target-name> can be also handled as
no match, ideally implementations document how they handle such
issues. The outcome for an unexpected <domain-spec> without macros
might differ from the outcome for an unexpected <target-name> after
macro expansion.
The last sentence in the Permerror definition is misleading. A
syntactically malformed <target-name> can be also handled as no
Be aware that it is also possible that this result is generated by
certain SPF clients due to the input arguments having an unexpected
format; see Section 4.8.
Note: Historically, this document has made no provisions for how to
handle <domain-spec>s, or macro-expansions thereof, that are
syntactically invalid per [RFC1035], such as names with empty labels
(e.g., "foo..example.com") or overlong labels (more than 63
characters). Some implementations choose to treat as a no-match
mechanisms, and ignore modifiers, with such names, whereas others
throw a "PermError" exception. The outcome for an unexpected
<domain-spec> without macros might even differ from that for an
unexpected <target-name> after macro expansion.
Reporting a TempError in such cases is no option, the syntax problem
won't go away for a given sender policy, HELO identity, envelope
sender address, and sending IP combination with a try again later
TempError approach. If anything else is as expected the sender policy
might need to be fixed manually, supporting PermError.
If the DNS syntax problem is caused by random net abuse, or
intentionally by an attacker, a no match approach, eventually
reaching a FAIL for -all or whatever the sender policy offers, is
better. In common practice this problem is handled as no match OR
PermError, like similar problems explicitly addressed in the
specification.

My preference here would be to treat these as PermError since an empty
domain-spec is not useful to the protocol and should be an error. I
think it's enough of a corner case that I don't think we need to warn
about the difference from RFC 4408. My solution (note: different than

A "PermError" result means that the domain's published records
could not be correctly interpreted. This signals an error
condition that requires manual intervention to be resolved, as
opposed to the TempError result. If the message is rejected
during the SMTP transaction for this reason, the software SHOULD
use an SMTP reply code of 550 and, if supported, the 5.5.2 DSN
code. Be aware that if the domain owner uses macros
(<xref target="macros"/>), it is possible that this result is due
to the checked identities having an unexpected format. The domain
and domain-spec MUST be fully macro expanded before being tested
for errors.

I don't think anything other than macro expansion before evaluation
makes sense. Please discuss.
Scott K
-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/
[http://www.listbox.com/member/]
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/1471225-766aee1f
https://www.listbox.com/member/?&
https://www.listbox.com/unsubscribe/?&&post_id=20111024163756:0CD3E470-FE80-11E0-89F0-A0E3D75F85EF
Powered by Listbox: http://www.listbox.com

[Frank Ellermann]

It appears to (rightly) push the issue of any
DNS specific syntactically invalid elements
introduced by a publisher / domain holder within
their zone file back to the relevant DNS related
RFC.

Makes sense, but IIRC there were two problems with
this approach: The reference implementation used
"no match" instead of PermError, i.e., there could
be a test case in the test suite that has to be
updated. Well, that's the main purpose of 4408bis,
get rid of the errata, ideally without introducing
new errors.

And it is not necessarily the fault of the policy.
As soon as the publisher tries to use local part
macros an attacker could try "interesting" local
parts such as "a..b"@example Is there any chance
to deprecate local part macros in 4408bis as known
trouble-makers?

-Frank

[Murray S. Kucherawy]

-----Original Message-----
Sent: Monday, October 24, 2011 10:05 PM
Subject: Re: [spf-discuss] RFC 4408 Errata - PermError on invalid domains after macro expansion
Is there any chance
to deprecate local part macros in 4408bis as known
trouble-makers?

If we are to apply typical IETF procedure for advancement of stuff up the standards track, then it's fine to remove stuff from a protocol if it can be shown that they're not in any kind of widespread use.

Does anyone have a survey of SPF records using local part macros versus total SPF records? If that ratio is very small, it's safe to remove.

[alan]

Post by Frank Ellermann
Is there any chance
to deprecate local part macros in 4408bis as known
trouble-makers?
-Frank
-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/13124949-0b42f103
Modify Your Subscription: https://www.listbox.com/member/?&
Unsubscribe Now: https://www.listbox.com/unsubscribe/?&&post_id=20111025010555:02DC1C8E-FEC7-11E0-AB8B-941D3167224D
Powered by Listbox: http://www.listbox.com

[Frank Ellermann]

Is there any chance to deprecate local part macros
in 4408bis as known trouble-makers?

I would hope not as my own and users domains use

Sigh, at least I tried it. It puts a burden on you
to allow only "sound" local parts working with all
(or most) 4408 implementations. And it will be very
interesting when you permit any "sound" UTF8SMTPBIS
local parts -- the IETF "EAI" RFCs are in Last Call.

So far I'm not aware of any SPF EAI implementations.

-Frank

[Stuart D Gathman]

Post by Frank Ellermann
Sigh, at least I tried it. It puts a burden on you
to allow only "sound" local parts working with all
(or most) 4408 implementations. And it will be very
interesting when you permit any "sound" UTF8SMTPBIS
local parts -- the IETF "EAI" RFCs are in Last Call.
So far I'm not aware of any SPF EAI implementations.

The "troublesome" localpart macro at worst causes PermError. For
v=spf3, I think (perhaps wrongly) that domain-spec should be UTF-8.
Failing that, we could add an option for the localpart macro to expand
non-ascii localparts as punycode. And exp= records should be
internationalizable, perhaps by assuming UTF-8 (of which US-ASCII, the
current 4408 requirement, is a subset).

[Murray S. Kucherawy]

-----Original Message-----
Sent: Monday, October 24, 2011 2:26 PM
Subject: Re: [spf-discuss] RFC 4408 Errata - PermError on invalid domains after macro expansion
Scott,
I think the second is better. It appears to (rightly) push the issue of
any DNS specific syntactically invalid elements introduced by a
publisher / domain holder within their zone file back to the relevant
DNS related RFC.
That seems more clear and appropriate.

+1.

[Alessandro Vesely]

Post by Scott Kitterman
My preference here would be to treat these as PermError since an empty
domain-spec is not useful to the protocol and should be an error.

Then PermError can result from both an incorrect SPF record and a
macro expanded illicit sender identity.

Post by Scott Kitterman
My solution (note: different than RFC4408 already due to other,

A "PermError" result means that the domain's published records
could not be correctly interpreted. This signals an error
condition that requires manual intervention to be resolved, as
opposed to the TempError result. If the message is rejected
during the SMTP transaction for this reason, the software SHOULD
use an SMTP reply code of 550 and, if supported, the 5.5.2 DSN
code.

I don't think it may gain wide consensus to reject on PermError. It
can result from inadvertently getting the syntax wrong, which is
totally unrelated with a third party acting maliciously. By removing
the last sentence above, it is more likely that the intended meaning
of the spec --reject on Fail-- will be conveyed.

Post by Scott Kitterman

Be aware that if the domain owner uses macros
(<xref target="macros"/>), it is possible that this result is due
to the checked identities having an unexpected format. The domain
and domain-spec MUST be fully macro expanded before being tested
for errors.

I'd put both these considerations in Section 8.1. In addition, the
note in the last but one paragraph there deserves an uppercase SHOULD,
before the explanation. E.g.

The domain and domain-spec MUST be fully macro expanded before
being tested for errors.

Note: Domains SHOULD avoid using the "s", "l", "o", or "h" macros
in conjunction with any mechanism directive. These macros can be
exploited so as to get a PermError instead of the intended outcome,
in certain circumstances: If the domain owner uses them, it is
possible that PermError results due to the checked identities
having an unexpected format. In addition, these macros severely
limit the ability of implementations to cache results of
check_host() and thus reduce the effectiveness of DNS caches.

jm2c

[Scott Kitterman]

Post by Alessandro Vesely

Post by Scott Kitterman
My preference here would be to treat these as PermError since an empty
domain-spec is not useful to the protocol and should be an error.

Then PermError can result from both an incorrect SPF record and a
macro expanded illicit sender identity.

Post by Scott Kitterman
My solution (note: different than RFC4408 already due to other,

A "PermError" result means that the domain's published records
could not be correctly interpreted. This signals an error
condition that requires manual intervention to be resolved, as
opposed to the TempError result. If the message is rejected
during the SMTP transaction for this reason, the software SHOULD
use an SMTP reply code of 550 and, if supported, the 5.5.2 DSN
code.

I don't think it may gain wide consensus to reject on PermError. It
can result from inadvertently getting the syntax wrong, which is
totally unrelated with a third party acting maliciously. By removing
the last sentence above, it is more likely that the intended meaning
of the spec --reject on Fail-- will be conveyed.

That's a confirmed errata (see the previous paragraph on TempError).
When the spec was written, the intent was to reject on PermError as
well. You may not agree with it, but your characterization of the
intent when it was written is not correct. In any case it does not
advise rejection, just gives advice about how to handle it if you do.

RFC 4408 deliberately steered clear of trying to specify receiver policy
(there is only one exception to this in the RFC). The receiver policy
(reject/don't reject) is a matter for local decision based on local
policy, but specifying how to handle such a rejection if such an option
is chosen is perfectly appropriate in the RFC.

[Murray S. Kucherawy]

-----Original Message-----
Sent: Tuesday, October 25, 2011 5:39 AM
Subject: Re: [spf-discuss] RFC 4408 Errata - PermError on invalid domains after macro expansion
RFC 4408 deliberately steered clear of trying to specify receiver policy
(there is only one exception to this in the RFC). The receiver policy
(reject/don't reject) is a matter for local decision based on local
policy, but specifying how to handle such a rejection if such an option
is chosen is perfectly appropriate in the RFC.

It's also fine to reference RFC5321 and whatever the enhanced status codes one is to back up the assertion of the SMTP reply code and enhanced code one has to use when choosing to reject. They're both standards track.

[Frank Ellermann]

Post by Scott Kitterman
That's a confirmed errata (see the previous
paragraph on TempError). When the spec was written,
the intent was to reject on PermError as well.  You
may not agree with it, but your characterization of
the intent when it was written is not correct.  In
any case it does not advise rejection, just
gives advice about how to handle it if you do.

+1 That was a rather controversial issue with lots
of misunderstandings, a formal appeal, an erratum +
figure out some consensus, etc., so please let's not
repeat this procedure:

The main thing is to _document_ the correct SMTP code
IF something is rejected. The spec. does nowhere say
that rejecting is REQUIRED, and IIRC it only goes so
far to say that rejecting FAIL is RECOMMENDED.

Your proposed text sounds fine. Of course you cannot
suggest two incompatible solutions for the same issue
in 4408bis; even if it's mostly a theoretical problem.

-Frank

[Alessandro Vesely]

Post by Frank Ellermann

Post by Scott Kitterman
That's a confirmed errata (see the previous
paragraph on TempError). When the spec was written,
the intent was to reject on PermError as well. You
may not agree with it but your characterization of
the intent when it was written is not correct.

Yes, I don't agree with that. That's why I surmised it wasn't the
intent of the spec. It would have made sense if it had specified,
say, StaticPermError (the RR) and DynamicPermError (the ids). I don't
think we can revert to such state now.

Post by Frank Ellermann
The main thing is to _document_ the correct SMTP code
IF something is rejected. The spec. does nowhere say
that rejecting is REQUIRED, and IIRC it only goes so
far to say that rejecting FAIL is RECOMMENDED.

Because "RFC 4408 deliberately steered clear of trying to specify
receiver policy", considering the possibility to reject is tantamount
to suggesting such behavior. I'd agree if it were documented clearly
and without false idiosyncrasies what is the intended semantics. For
example, the text could say

If the message is rejected during the SMTP transaction because of
this reason --which is NOT RECOMMENDED-- then the software SHOULD
use an SMTP reply code of 550 and, if supported, the 5.5.2 DSN
code.

Better yet, I would add to /each/ subsection 2.5.1..7 a pedantically
prominent statement of the form

The checking MTA <RFC 2119 KEY WORD> reject an SMTP transaction
based [solely] on this result / unless there are other reasons.

I'd propose to check what's the consensus on this. Just how far we
are from one another, after six years[*]. Obviously, it would be much
better to know how actual MTAs are configured, but I have no idea how
to run such survey. The following are IMHO-values (that is, how I
think they are in the wild --not how I'd like them to be documented):

2.5.1. None . . . . MUST NOT
2.5.2. Neutral . . MUST NOT
2.5.3. Pass . . . . MUST NOT
2.5.4. Fail . . . . SHOULD
2.5.5. SoftFail . . SHOULD NOT
2.5.6. TempError . MUST NOT (but MAY reply 4xx)
2.5.7. PermError . SHOULD NOT

Section 1.3 mentions 10 key words, so there are 10^7 total
possibilities, but I'd expect no more than two or three will be upheld
by multiple participants.

[Frank Ellermann]

Post by Alessandro Vesely
Because "RFC 4408 deliberately steered clear of
trying to specify receiver policy", considering
the possibility to reject is tantamount to
suggesting such behavior.

Well, it was specified in one or more drafts, it
somehow vanished in the RFC, I appealed that, the
Council at this time confirmed that it should be
documented, it got an erratum, and IIRC another
Council confirmed the erratum.

Any step with long threads on this list. The RFC
doesn't exactly tell you what to do with SOFTFAIL,
TEMPERROR, PERMERROR, or NEUTRAL (of course you're
free to reject SPF PASS if it's a known spammer or
similar, and you are also free to accept FAIL if
you have a good reason to ignore the only "SHOULD"
in this context, because this could result in lost
mails if you don't know the consequences in your
setup - junk folder - purge - oops - the works).

For the more interesting cases the spec. tells you
how to reject it *IF* you reject it, for FAIL, for
SOFTFAIL, and for TEMPERROR. Omitting this only
for PERMERROR always was an error. It is a simple
interop consideration, everybody picking their own
idea of SMTP + DSN code can't be good.

If that is irrelevant for you because you'd never
reject PERMERROR simply ignore the SMTP + DSN code.

-Frank

P.S. (unrelated): Murray + Hector try to document
Graylisting on the SMTP list in different drafts.

[Alessandro Vesely]

Post by Alessandro Vesely
Because "RFC 4408 deliberately steered clear of trying to specify
receiver policy", considering the possibility to reject is
tantamount to suggesting such behavior.

Any step with long threads on this list. The RFC doesn't exactly
tell you what to do with SOFTFAIL, TEMPERROR, PERMERROR, or NEUTRAL
(of course you're free to reject SPF PASS if it's a known spammer
or similar, and you are also free to accept FAIL [...])

This somehow assumes that "receiver policy" and "sender policy" are
two unrelated areas. Yes, it is a recurrent topic. Some time ago
Julian wrote

The problem with mandating receiver policy is that receivers are
going to ignore it at will. Receivers will always do what they
think is best for them.

That is obviously true of any IETF standard. However, in order to let
senders know whether they're safe using macros, in case we happen to
know that senders should not reject on PermError, we'd better document
that.

For the more interesting cases the spec. tells you how to reject it
*IF* you reject it, for FAIL, for SOFTFAIL, and for TEMPERROR.
Omitting this only for PERMERROR always was an error.

It is an ambiguous way to tell which cases are interesting. And
"MAY", "SHOULD", or "MUST" are much clearer than "interesting" anyway.
(Or perhaps we could explain in greater detail how to reject for the
cases that are more interesting :-)

jm2c

[Scott Kitterman]

Post by Alessandro Vesely

Post by Alessandro Vesely
Because "RFC 4408 deliberately steered clear of trying to specify
receiver policy", considering the possibility to reject is
tantamount to suggesting such behavior.

Any step with long threads on this list. The RFC doesn't exactly
tell you what to do with SOFTFAIL, TEMPERROR, PERMERROR, or NEUTRAL
(of course you're free to reject SPF PASS if it's a known spammer
or similar, and you are also free to accept FAIL [...])

This somehow assumes that "receiver policy" and "sender policy" are
two unrelated areas. Yes, it is a recurrent topic. Some time ago
The problem with mandating receiver policy is that receivers are
going to ignore it at will. Receivers will always do what they
think is best for them.
That is obviously true of any IETF standard. However, in order to let
senders know whether they're safe using macros, in case we happen to
know that senders should not reject on PermError, we'd better document
that.

We need to be careful not to over-specify. The RFC needs to focus on
the bits needed for interoperability. It is not a full implementation
spec (although RFC 4408 manages to come close to this).

For the question of what to do with invalid domains after macro
expansion, for purposes of interoperability, we ought to specify the SPF
result that is appropriate. What receivers do with that result is not a
matter of interoperability, but of local policy.

We went through this repeatedly in 2004 - 2006 and came to a rough
consensus that specifying receiver policy would be a mistake. I don't
think that is going to change.

Senders are never 'safe'. There is never a guarantee that receivers
will accept their mail, so trying to bend the standard in this one case
doesn't help.

In the pre-MARID specs what is now known as permerror was simply called
unknown. In the work that was done in MARID and after there was a
deliberate design choice to call this permerror and that rejecting based
on it was a possibility. Right or wrong, I think your ship sailed 5
years ago. The fact that the sentence about exactly how to reject if
one chose to do so was a simple editorial mistake.

Post by Alessandro Vesely

For the more interesting cases the spec. tells you how to reject it
*IF* you reject it, for FAIL, for SOFTFAIL, and for TEMPERROR.
Omitting this only for PERMERROR always was an error.

It is an ambiguous way to tell which cases are interesting. And
"MAY", "SHOULD", or "MUST" are much clearer than "interesting" anyway.
(Or perhaps we could explain in greater detail how to reject for the
cases that are more interesting :-)

Paragraph 2.5.4 already covers the more interesting reject case.

Scott K