Browseable Test Harness

Dominique Hazael-Massieux has created a CGI-based harness for the Mobile Web Initiative's test suites. We plan to extend this harness and use it for the CSS conformance tests and implementation reports. One of the very neat things about the MWI test harness is that it associates the results with a user agent string. This means there's no need for users to log in or to select their UA. They just load a test and click Pass/Fail/Can't Tell.

See discussion on public-css-testsuite.

• MWI's harness
• MWI's harness source code
• New harness prototype
• New harness source code
• New harness bug database

These are the improvements we'd like to see, numbered roughly by priority.

• (1) Harness using <object> to contain the test and pass/fail buttons on the containing page rather than inside the test file. (This format is good for browsers.) The <object> will contain a hypertext link to the test as fallback.

• (2) Harness using links to open the test and pass/fail buttons on the page containing the link rather than inside the test file. This format will also allow the tester to enter the UA string manually, for cases where the UA being tested is non-interactive. (This format is necessary for printers.)

In both cases the test itself should be referenced as a link, not fed through the CGI script. (This avoids tampering with the HTTP headers that normally get served up with the tests.)

• (8) It would be nice if these harnesses could include some meta information about the test in addition to the buttons. E.g. the test ID (filename before extension), test title, any requirements documented in the test (“Warning: Must install Ahem font.” etc). (We can extract this information into a flat-file database during the test suite build process so the harness doesn't have to do any analysis of the tests themselves.)

• (3) Ability to consolidate results for various user agent strings under one category name. E.g. consolidate results for all UA strings that represent Opera 9.25 Beta 1 regardless of OS and localization.

• (4) Pass/fail scores for the whole test suite

• (5) Ability to report consolidated pass/fail scores for a named groups of tests. (To create e.g. a summary of what features are supported and to what level.)

• (6) Interface for generating reports based on various parameters. URLs to these reports should be short and clean so they can be passed around in blogs/IM/email etc.

• (9) Prettier reports. :)

• (7) Ability to delete all data for a given test (so that when a test is changed we can invalidate the results for that test). This process could be triggered manually on the command line rather than via CGI—that avoids the need for a login system.

• Put requirements up front
• Add Skip button
• Track session so that a bad user's data can be deleted