Case-sensitive status

Hi Murray!

Post by Murray S. Kucherawy
I'm aware that Scott has started the work of putting together a push to
move SPF from an Experimental to Standards Track document within the
IETF. I'll be helping out where I can.

Definitely a worthwhile effort! I've seen your recent thread on

Post by Murray S. Kucherawy
One change I'd like to suggest has to do with the SPF results strings.
RFC4408 presents these status strings in a specific case. RFC5451
presents them in all-lowercase. Neither one explicitly says anything
about whether or not consumers of the results need to test them in a
case-sensitive manner, although in a roundabout way they're
case-insensitive because ABNF (RFC5234) says so when case-sensitivity
isn't expressly stated.
Would it break any known implementations to change them to
all-lowercase in the new RFC, just to be consistent with other things?

I read RFC 4234 (the predecessor revision of RFC 5234) before I went and
wrote the Mail::SPF Perl module under the assumption that the result
codes would be interpreted case-*in*sensitively exactly because RFC 4234,
section 2.3, "Terminal Values", said they would. Of course Mail::SPF is
not a *consumer* of SPF result codes, but only a producer (of Received-
SPF headers).

The most significant consumer of SPF result codes I can think of probably
is SpamAssassin. There are certainly others.

From my experience I strongly doubt any significant software packages
consuming SPF result codes do so in a case-sensitive manner, but of
course that's only a gut feeling.

Murray S. Kucherawy

2011-08-09 21:00:25 UTC

-----Original Message-----
Sent: Tuesday, August 09, 2011 1:54 PM
Subject: [spf-discuss] Re: Case-sensitive status
From my experience I strongly doubt any significant software packages
consuming SPF result codes do so in a case-sensitive manner, but of
course that's only a gut feeling.

Anybody from SA, or someone familiar with its code, on the list that can confirm this?

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/6959934-50ec8f89
Modify Your Subscription: https://www.listbox.com/member/?member_id=6959934&id_secret=6959934-b7c4528d
Unsubscribe Now: https://www.listbox.com/unsubscribe/?member_id=6959934&id_secret=6959934-edadf31a&post_id=20110809170038:9E8196CA-C2CA-11E0-9C1F-D6682B7734FC
Powered by Listbox: http://www.listbox.co

Julian Mehnle

2011-08-09 21:51:51 UTC

Post by Julian Mehnle
From my experience I strongly doubt any significant software packages
consuming SPF result codes do so in a case-sensitive manner, but of
course that's only a gut feeling.

Anybody from SA, or someone familiar with its code, on the list that can confirm this?

I'm not familiar with the SpamAssassin code, but I looked anyway:

http://svn.apache.org/viewvc/spamassassin/trunk/lib/Mail/SpamAssassin/Plugin/SPF.pm?view=markup#l336

See the //i regexp option in line 355? It matches case-insensitively.

That is also the only file in the SA code base (as of 3.3.2) consuming
Received-SPF headers.

- -Julian

Scott Kitterman

2011-08-09 22:31:26 UTC

Post by Julian Mehnle
From my experience I strongly doubt any significant software packages
consuming SPF result codes do so in a case-sensitive manner, but of
course that's only a gut feeling.

Anybody from SA, or someone familiar with its code, on the list that can confirm this?

http://svn.apache.org/viewvc/spamassassin/trunk/lib/Mail/SpamAssassin/Plug
in/SPF.pm?view=markup#l336
See the //i regexp option in line 355? It matches case-insensitively.
That is also the only file in the SA code base (as of 3.3.2) consuming
Received-SPF headers.

I checked pypolicyd-spf, which is the Postfix policy server for SPF I developed
and it does not make any assumptions about the case of the result code (it
downcases and then sets appropriate case to give a consistent presentation in
SPF Received headers, but that would be trivial to change).

I think it should be fine.

Scott K

G.W. Haywood

2011-08-10 08:59:41 UTC

Hi there,

Just wondering if this talk about case impinges on e.g. non-Western
character sets in any way.

--

73,
Ged.

Julian Mehnle

2011-08-10 16:10:55 UTC

Post by G.W. Haywood
Just wondering if this talk about case impinges on e.g. non-Western
character sets in any way.

No. *Everything* is US-ASCII in SPF.

- -Julian

Stuart D. Gathman

2011-08-10 20:47:06 UTC

Post by G.W. Haywood
Just wondering if this talk about case impinges on e.g. non-Western
character sets in any way.

No. *Everything* is US-ASCII in SPF.

We ought to define an encoding to use for TXT records referenced by exp=

--
Stuart D. Gathman <***@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Scott Kitterman

2011-08-10 21:33:23 UTC

Post by Stuart D. Gathman

Post by G.W. Haywood
Just wondering if this talk about case impinges on e.g. non-Western
character sets in any way.

No. *Everything* is US-ASCII in SPF.

We ought to define an encoding to use for TXT records referenced by exp=

I think that's SPF v 3 material.

Scott K

Scott Kitterman

2011-08-10 21:51:58 UTC

Post by Scott Kitterman

Post by Stuart D. Gathman

Post by G.W. Haywood
Just wondering if this talk about case impinges on e.g. non-Western
character sets in any way.

No. *Everything* is US-ASCII in SPF.

We ought to define an encoding to use for TXT records referenced by exp=

I think that's SPF v 3 material.

Actually, since exp is a modifier, you could propose a new one, like exp-
encoded= and write an ID to define it.

Scott K

Frank Ellermann

2011-08-12 01:36:45 UTC

in a roundabout way they’re case-insensitive because ABNF (RFC5234) says so when
case-sensitivity isn’t expressly stated.

I use your mail as found in the list archive to test that I can in
fact send mail
to the list, threading will likely fail. Yes, ABNF is case-insensitive unless
folks go to the trouble and specify cases-sensitive strings in hex. notation.

Would it break any known implementations to change them to all-lowercase in the
new RFC, just to be consistent with other things?

It can't, because ABNF is case-insensitive ;-) There should be test
cases in the
test suite covering this point, in other words, it's mostly a question
of what is
better readable in 4408bis. In the prose I'd prefer the mixed case
"as is", but
wouldn't waste time with arguing if you think that this is confusing.
In the ABNF
I don't care, lower case instead of camel case is fine.

That part of the ABNF apparently only affects the Received-SPF header
field. Is
there any chance to get rid of this in 4408bis, e.g., with a pointer
to RFC 4408,
plus some "please use RFC 5451" blurb? Any "erratum" for Received-SPF should be
listed, I vaguely recall that this is the case, but this could be buried in some
"things you really do not more need to care about" appendix.

-Frank

Murray S. Kucherawy

2011-08-12 03:55:08 UTC

-----Original Message-----
Sent: Thursday, August 11, 2011 6:37 PM
Subject: [spf-discuss] Re: Case-sensitive status

Would it break any known implementations to change them to all- lowercase in the
new RFC, just to be consistent with other things?

It can't, because ABNF is case-insensitive ;-)

That doesn't mean people did it right. :-)

There should be test cases in the
test suite covering this point, in other words, it's mostly a question of what is
better readable in 4408bis. In the prose I'd prefer the mixed case "as is", but
wouldn't waste time with arguing if you think that this is confusing. In the ABNF
I don't care, lower case instead of camel case is fine.

I think there's a risk people will/did implement it verbatim, missing the nuance that ABNF string comparisons are case-insensitive unless actual octets are specified.

That part of the ABNF apparently only affects the Received-SPF header field. Is
there any chance to get rid of this in 4408bis, e.g., with a pointer to RFC 4408,
plus some "please use RFC 5451" blurb? Any "erratum" for Received-SPF should be
listed, I vaguely recall that this is the case, but this could be buried in some
"things you really do not more need to care about" appendix.

I just replied to this on apps-discuss, but basically if there's some reasonable demonstration that the world has largely dropped Received-SPF in favour of Authentication-Results, then such a change should probably be fine.

-MSK

Frank Ellermann

2011-08-12 04:20:27 UTC

Post by Murray S. Kucherawy
basically if there's some reasonable demonstration that the world has
largely dropped Received-SPF in favour of Authentication-Results, then
such a change should probably be fine.

Sadly I can't produce that, it must be hidden in the ominously missing
[2009] entry on <URL:http://www.openspf.org/auth/Research>. I get mails
from the SPF "discuss" list and can edit SPF wiki pages, but I have not
yet checked if I perhaps missed any "nobody uses Received-SPF anymore"
info.

Seriously, okay, I can forget any vague idea to smuggle EAI into 4408bis
even if that would be possible (unlikely).

-Frank

Julian Mehnle

2011-08-12 04:44:00 UTC

Post by Frank Ellermann

Sadly I can't produce that, it must be hidden in the ominously missing
[2009] entry on <URL:http://www.openspf.org/auth/Research>. [...]

For the benefit of all, that is:

http://www.openspf.org/Research

(Remember to remove the "auth/" part from those URLs.)

Frank Ellermann

2011-08-12 04:52:20 UTC

Post by Julian Mehnle
(Remember to remove the "auth/" part from those URLs.)

ACK, I vaguely recall that this ends up on a page explaining what
went wrong. Too bad that Received-SPF isn't covered in the test
suite, but your (plural) proposals how to tackle it with a SHOULD
in 4408bis are good.

-Frank

Julian Mehnle

2011-08-12 04:40:51 UTC

Post by Murray S. Kucherawy
Would it break any known implementations to change them to all-
lowercase in the new RFC, just to be consistent with other things?

[...]

There should be test cases in the test suite covering this point,

No, the RFC 4408 test suite doesn't test for "Received-SPF" header
generation (which is the only aspect of SPF that *outputs* result names).

[...]

I think there's a risk people will/did implement it verbatim, missing
the nuance that ABNF string comparisons are case-insensitive unless
actual octets are specified.

The possibility certainly exists, however there's also the "be tolerant in
what you accept" principle, which probably caused most implementors to
handle result names case insensitively.

That part of the ABNF apparently only affects the Received-SPF header
field. Is there any chance to get rid of this in 4408bis, e.g., with
a pointer to RFC 4408, plus some "please use RFC 5451" blurb? Any
"erratum" for Received-SPF should be listed, I vaguely recall that
this is the case, but this could be buried in some "things you really
do not more need to care about" appendix.

I just replied to this on apps-discuss, but basically if there's some
reasonable demonstration that the world has largely dropped
Received-SPF in favour of Authentication-Results, then such a change
should probably be fine.

I think the best way to deal with the "Received-SPF" header in 4408bis is
to define its grammar for implementors who would like to *parse* it, but
then say that it SHOULD NOT be generated and that "Authentication-Results"
SHOULD be generated instead.

Alessandro Vesely

2011-08-13 18:07:18 UTC

Post by Julian Mehnle
I think the best way to deal with the "Received-SPF" header in 4408bis is
to define its grammar for implementors who would like to *parse* it, but
then say that it SHOULD NOT be generated and that "Authentication-Results"
SHOULD be generated instead.

I'd be even more conservative. That is, if an application handles A-R
fields, then it should produce those instead of Received-SPF. Otherwise, it
may use the latter to communicate results to a downstream agent. This way,
programmers should read RFC 5451 if they want to implement it, not just
blindly replace a header field with another.

jm2c

Frank Ellermann

2011-08-13 18:44:46 UTC

Post by Alessandro Vesely
I'd be even more conservative.

Good point, Authentication-Results in conflict with Received-SPF
would indicate that something might be not as it should be.

It could be a "sound" side effect of different DNS queries while
processing the same mail, but more likely this is simply a "bug".

-Frank

Alessandro Vesely

2011-08-14 08:09:48 UTC

Hi Frank!

Post by Frank Ellermann

Post by Alessandro Vesely
I'd be even more conservative.

Good point, Authentication-Results in conflict with Received-SPF
would indicate that something might be not as it should be.

Lack of inter-agent communication, unless Received-SPF had a TempError.

For an example, Courier-MTA doesn't put an A-R field, but puts Received-SPF
if properly configured; a filter that implements RFC 5451 can ease its job by
reading any existing Received-SPF. E.g., zdkimfilter operates this way,
without implementing check_host, but possibly adding other results.

Implementing RFC 5451 requires to choose an "authserv-id" and to filter any
existing, possibly spurious A-R fields that bear the same token. It is no
rocket science, but may need to be coordinated with any existing agent that
already sets A-R fields (most commonly, for DKIM / DomainKeys checks.)

Scott Kitterman

2011-08-12 04:08:05 UTC