This is an informal schema, illustrated with examples. The structure is agreed upon. A formal schema is underway, and the council will vote on naming issues if necessary.
The syntax is YAML. The top level object is a "scenario". A file can consist of multiple scenarios separated by '---' on a line by itself. Lexical comments are introduced by '#' and continue to the end of a line. Lexical comments are ignored. There are also comment fields which are part of a scenario, and used for purposes such as automating the annotated RFC. Here are two sample scenarios:
# Sample scenarios from pyspf test suite.
description: >-
check trailing dot with redirect and exp
tests:
traildot1:
spec: 8.1
description: Check that trailing dot is accepted for domains
host: 192.168.218.40
helo: msgbas2x.cos.example.com
mailfrom: test@example.com
result: pass
traildot2:
spec: 8.1
description: Check that trailing dot is not removed from explanation
helo: msgbas2x.cos.example.com
host: 192.168.218.40
mailfrom: test@exp.example.com
result: fail
explanation: This is a test.
zonedata:
example.com.d.spf.example.com:
- SPF: v=spf1 redirect=a.spf.example.com
a.spf.example.com:
- SPF: >-
v=spf1 mx:example.com include:o.spf.example.com -exists:%{s}.S.bl.spf.example.com
exists:%{s}.S.%{i}.AI.spf.example.com ~all
o.spf.example.com:
- SPF: v=spf1 ip4:192.168.144.41 ip4:192.168.218.40 ip4:192.168.218.41
msgbas1x.cos.example.com:
- A: 192.168.240.36
example.com:
- A: 192.168.90.76
- SPF: v=spf1 redirect=%{d}.d.spf.example.com.
- MX: [10, msgbas1x.cos.example.com]
exp.example.com:
- SPF: v=spf1 exp=msg.example.com. -all
msg.example.com:
- TXT: This is a test.
---
description: test empty MX
tests:
emptyMX:
helo: mail.example.com
host: 1.2.3.4
mailfrom: ''
result: neutral
zonedata:
mail.example.com:
- MX: [0, '']
- SPF: v=spf1 mx
A scenario has four attributes: zonedata, tests, description, comment. description is a free form multi-line string. tests and zonedata are maps. comment is additional commentary about the test of less importance than the description.
zonedata
Each key in the zonedata map is a DNS name, and its value is a list of maps, each with a single entry. The key of each single entry map is the DNS record type as text (upper case prefered). The value is a string or list. If the value of an SPF or TXT record is a string, it is promoted to a list of length 1. Note that the value of SPF or TXT is always a list of strings.
DNS errors
When the last entry for a DNS name is the string TIMEOUT, the test driver should simulate a DNS timeout exception for queries that do not match any preceding records. When the last map for a DNS name is the map RCODE: n, the test driver should simulate a DNS error with code n. The test suite currently uses TIMEOUT, but not RCODE.
Records of type SPF get special treatment. If no records of type TXT are given for the same DNS name, then an identical TXT record is also generated for the DNS data. This reflects the recommendation of section 3.1.1 and allows the test suite to be used with implementations that choose any of the three options in section 4.4. In addition, when the value of an SPF name is the string NONE, then that record is not added to the DNS data. As a result, TXT: NONE serves to suppress the auto copy of SPF records to TXT. This allows testing of record selection rules.
tests
Each key in the tests map is the name of a test case. All the test cases in the scenario will use the DNS records provided in zonedata, and *only* those records. Your test driver should load the DNS records, and hook DNS lookups for your SPF library to use those records. If needed, you can load the records into a DNS server under a subdomain you control, for example scenario1._spftest.example.com, and arrange to append that subdomain to all lookups within your SPF library. A key design goal is for DNS data used by the tests to be contained within the test specification.
Each test case has four required attributes, helo, host, mailfrom, result. helo is an arbitrary string, and not necessarily a legal rfc2821 helo name. Remember, spammers and clueless senders put whatever garbage they want in HELO. host is an IP4 address in dotted decimal notation, or an IPV6 address in standard notation. If your implementation does not handle IP6, it should ignore test cases where host is not an IP4 address. Such a library should never get an IP6 connection. Note that even an IP4 only library must handle some IP6 syntax elements, for example dual-cidr-length. mailfrom is an arbitrary string. Again, your library must be able to handle garbage.
result is one of the defined SPF results as a lower case string. Your test driver should call your SPF library with the basic test case data, and verify that your library gets the expected SPF result. In some cases, because of ambiguity in RFC4408, there are multiple results. Multiple results are supplied as a list. The first result in the list is the "preferred" result, and is the result required according the majority interpretation of the RFC. Your test driver should print a warning if the library result is in the list, but is not the preferred result. You can, of course, ignore the warning if you disagree with the majority.
Optional attributes for test cases are spec, description, comment, and explanation. spec is a reference to section and paragraph of RFC 4408. It is a simple string, or a list of strings if there are multiple references. The format of spec references is section/paragraph/sentence, where paragraph and sentence are optional. Paragraphs are numbered using the official IETF text (despite some formatting errors) counting each block separated by blank lines or list of ABNF rules (ignoring page breaks) as a paragraph. For example, section 8.1 has 29 paragraphs, and 8.1/5 reads:
The following macro letters are expanded in term arguments:
If explanation is provided, and your implementation supports the exp modifier, then the test driver should verify that the failure explanation returned by your library matches. If the explanation attribute is the magic string DEFAULT, then the test driver should verify that the default explanation was returned. If the implementation supports setting the default explanation, this is easily accomplished by setting the default explanation to "DEFAULT". The test driver should ignore the explanation attribute for implementations that do not support the exp modifier.
Each test case should provide description as a brief (one or two sentence) description of what is being tested. The comment attribute should be used for an extended explanation.
Sample Driver
The python test driver used for pyspf may be helpful in creating drivers for other SPF libraries.
