#20: A good reference implementation and test suite shall be maintained
A reference implementation and a test suite shall be maintained. They shall both strictly adhere to the SPF specification and be of high quality.
The reference implementation
Council member Julian Mehnle is appointed to the creation and maintenance of the reference implementation, and may call in the community for help if necessary.
The Perl module Mail::SPF::Query is the current reference implementation, even though it has some deficiencies, some of which will be addressed.
Generally, non-specified features such as best guess processing or trusted forwarder accreditation checking may be included with the reference implementation as long as they are clearly marked as non-standard in the code and documentation and are disabled by default. If at all possible, they should be included in a form that is clearly separate from the core implementation.
The test suite
Council member Wayne Schlitt is appointed to the creation and maintenance of the test suite, and may call in the community for help if necessary.
(Proposed on 2005-01-15 by Julian Mehnle and Wayne Schlitt, passed by majority vote. First amendment proposed on 2005-11-20 by Greg Connor, passed by majority vote.)
<csm> The SPF reference implementation and the test suite should be
<csm> updated. Who can do it?
<Julian> SPF reference implementation == Mail::SPF::Query
<grumpy> Well, so says spf.pobox.com....
<Julian> Do you agree that it should be a priority?
<grumpy> I don't think it makes a particularly good reference
implementation. I'm not sure that any of the implementations do.
<MarkK> I do not feel comfortable enough in IPv6 to do it
<grumpy> do we need a reference implementation?
<Julian> I guess I could do it, but I have too many other things on my SPF
TODO list.
<MarkK> yes, we need a reference implementation
<grumpy> isn't it time to move to a spec-based reference?
<Julian> grumpy: Maybe, but a stable and mostly bug-free reference
implementation would indeed very good to have.
<grumpy> MarkK: why? What's the reference implementation for SMTP?
<MarkK> I really think ppl need a piece of software to model their
implementation-behavior agyer
<Julian> Existing code is one of SPF's assets.
<MarkK> after, even
<grumpy> So, if the "reference" implementation and the spec disagree, which
is right?
<Julian> grumpy: The spec.
<MarkK> grumpy: because new coders want to test their software input/output
against reference code;
<grumpy> I think they should test it against a test suite
<grumpy> if anything.
<Julian> grumpy: But just like we have to take care that the spec doesn't
contradict itself, we can take care that the reference
implementation doesn't contradict the spec.
<Julian> grumpy: A test suite would be good, yes. We could test the ref.
impl. against it.
<grumpy> shouldn't all implementations conform to the spec?
<Julian> grumpy: Sure.
<grumpy> why do we need a single reference implementation?
<MarkK> grumpy: sure; but when I write code, I always like to find a
reference implementation too, just to make just my code behaves as
expected; that is not always clear from a spect alone
<grumpy> and if the reference implementation has features that aren't in the
spec, does that mean everyone else needs to have that feature?
<Julian> grumpy: We can have many if we want. But there should at least be
an official one that can be studied by authors of other
implementations.
<Julian> grumpy: The ref. impl. should not have features that are not in the
spec.
<grumpy> Well, a *good* reference implementation has few, if any, additional
features and everything is written to be clear and correct, rather
than fast
<MarkK> julian: agreed; it is also politically sound, as it lowers the
threshold for new coders (and thus promotes SPF adoption)
<grumpy> none of the SPF implementations come close to that.
<Julian> At least, additional features should be modular and one should be
able to disable them easily.
<grumpy> I think a good test suite is more important
<Julian> grumpy: Ok, I won't object.
<Julian> But a good ref. impl. is not unimportant.
<MarkK> I do not think it needs to be either/or; a good test-suite is very
important too
<Julian> Alright, I can do a new reference implementation, but it will take
a while (~2 months or so).
<grumpy> Well, I think the priorities should be: 1) a spec 2) a test
suite 3) practical implementations that pass the spec and test
suite 4) ref impl
<MarkK> julian: that would be more than adequate time; it has no hurry, but
should, at some point, really be done
<Julian> We could solicit someone from the community, too.
<MarkK> grumpy: agreed on that priority
<Julian> grumpy: Swap 3 and 4 and I can agree. :-)
<Julian> If we have 4, we probably don't need 3 anymore. But having 3 makes
getting 4 easier.
<grumpy> Anyway, I *don't* think that M:S:Q makes a good reference
implementation
<MarkK> both 3 and 4 can be 3, really, ihmo (as having more or less the
same importance to me)
<Julian> Ok, what about 2?
<Julian> Who can make the test suite?
<grumpy> Well, I did the last one
<csm> if anyone fixes anything I hope that we can get the perl stuff for
postfix fixed and get it all tohgether in one place like I have
done at http://moongroup.com/downloads/
<Julian> Is it up to date?
<grumpy> I can continue to work on it, but I think that needs to wait until
we have a final spec
<grumpy> it is pretty close to 200406
<grumpy> it tests things like unknown mechanisms, which have changed since
200406 though
<grumpy> The real problem is I was the only one that really ended up using
it.
<grumpy> So, what's the point?
<Julian> It's a chicken/egg problem.
<grumpy> (Ok, the point for me was to test my implementation, but...)
<Julian> If you think a test suite is important, then we should do one.
<grumpy> Ok, when I get done with the spec and update my implementation to
conform to the lastest spec, I'll update the test suite.
<Julian> Ok, motion 1: Wayne shall take care of making a good test suite,
calling in the community for help if necessary.
<Julian> Motion 2: Julian shall take care of making a good reference
implementation, calling in the community for help if necessary.
<Julian> (No time frames.)
<grumpy> seconded
<csm> votes
<Julian> 2031u: yes
<grumpy> 2031: yes
<MarkK> 2031u: yes
<csm> so ordered
Vote log for 2005-11-20 amendment | >> |
<gconnor_l> modified motion: Yes, there should be a reference implementation.
Right now it is M::S::Q, though there are some bugs, which should
be addressed. There may be another reference implementation in
the future. Extra stuff is allowed at the discretion of the
author/maintainer, as long as it is turned off and performs per
spec by default. In that case the documentation should clearly
say that the feature is non-standard and may not produce correct
results"
<Julian> 01:07u: seconded.
<MarkK> gconner: make that: "as long as it is turned off by default"
<gconnor_l> Yes, that's what I meant
<Julian> MarkK: That's implied.
<grumpy> I disagree that M:S:Q can be a reference implemntation right now.
<grumpy> it has been broken in many ways for a long time.
<gconnor_l> "as long as it is turned off and >the code< performs per spec by
default"
<gconnor_l> grumpy- In that case we have nothing to try and agree on right
now.
<MarkK> yes, we might wrap up for the day
<grumpy> gconnor_l: I think I can agree with that.
<Julian> grumpy: Look, it's not as if we were going to send out an
announcement on spf-announce that "M:S:Q is still our reference
implementation" and that lots of people would suddenly start
using M:S:Q as a coding reference now.
<gconnor_l> I thought it was the de-facto reference code, but if that's not
what other people already think, I'm probably off base
<MarkK> I think it is still presented as such
<grumpy> yes, it is still presented as such.
<MarkK> but not actively. :)
<gconnor_l> To the extent it is presented as such (in documents, etc) it
should have disclaimers added.
<Julian> The resolution we're going to make now is not about changing the
outside (outside the project) view on whether M:S:Q is a (good)
reference implementation. It is about clarifying what should
happen with some of the features of M:S:Q, which without question
_is_ the de-facto reference implementation.
<Julian> We can't change the latter part through whatever declarations
now. And this isn't the point of this agenda item anyway.
<Julian> The point of this agenda item is to decide what should happen
with M:S:Q now.
<freeside> the non-core stuff like best_guess and whatnot are all turned off
anyway ... why not leave them in, and optional?
<Julian> So, can we vote on 0107u now, as I have seconded it?
<Julian> freeside: You're absolutely right.
<freeside> i mean, we want to encourage trusted_forwarder usage, and putting
it in there, but turned off, helps encourage that.
<grumpy> Ok, let's vote on 0107u...
<grumpy> votes on 01:07u?
<MarkK> I already suggested my change, to remove "/24" from best-guess
(and to apply bug-fix patch). Other than that, I'm fione with its
current incarnation.
<Julian> 01:07u: yes
<gconnor_l> 01:07u: yes
<grumpy> 0107u: no
<freeside> 0107u: yes
<Julian> Thank you. Thanks to gconnor_l, in particular.
<Julian> (for the motion)
<freeside> and i suggest we aim to bugfix msq as quickly as possible so that
grumpy can feel comfortable with it being a reference
implementation.
<grumpy> MarkK?
<MarkK> 0107u: yes
<grumpy> ok, motion passes.