[Members] First XSF Certificate Meeting

Nathan Fritz nathanfritz at gmail.com
Mon May 11 17:37:30 CDT 2009


> "Certificate" makes me think of X.509. Perhaps we could call it
> "Certification" instead.


What's in a name?  Sure, certification is fine.

Another action item would be to research open-source testing tools.
> Perhaps http://www.opensourcetesting.org/ has some useful links.
>
> It seems to me that we'd want to develop an extensible approach that
> enables XSF members and interested others to write their own test sets
> for XMPP functionality. Something scriptable comes to mind. :)
>

Sure, I think that expecting everyone to contribute to a single harness in a
single language would be ideal but is a fantasy.  I'll look into
opensourcetesting.org.  Perhaps expecting specific result from a test
harness would allow us to span all of the languages that our members enjoy,
but still us to have a unified interface.

For example, our harness could run as a webapp, and call individually
contributed tests (in various languages) calling each with a standardized
configuration XML format, and returning a standardized XML result so that
the webapp could display and total the results from disparate tests in a
unified way.  This seems like a "down the road" idea, and I think we'd be
better served by starting without at first.  We could work up to it.


> What's the general approach behind your test harness?
>
>
Here's something I wrote to explain it to someone else:

XMPP Publish-Subscribe Test Harness
>
> This test harness is designed to assist QA staff in finding bugs, provide
> developers with clear goals and tasks to develop against, and build customer
> confidence in specification compliance and stability.  The harness is
> available via command line or website, is cross platform, can email detailed
> results, and be automated with a check-in procedure.  A configuration file
> determines credentials for JIDs to test from, values to test against (for
> example, a size of pubsub payload that should cause an “payload too large
> error”) and other values.  The test will be scored based on various
> compliance levels, as well as a raw pass/fail ratios.  This harness is
> designed to be applied to other areas of XMPP compliance testing, but is not
> a generic test suite for any other wire protocol.
>
> The following tests will be applied to *each feature* in XEP-0060:
>
>    - Tests that check for proper error responses.
>    - DTD compliance tests for all incoming and outgoing traffic during
>    each test.
>    - Tests from local accounts, federated accounts, and SASL-Anonymous
>    bound JIDs
>    - Tests for each node affiliation against every applicable feature.
>    - Tests that compare bareJIDs, fullJIDs, and federated, local, and
>    anonymous JIDs in every combination reasonable against every JID field for
>    every applicable feature. This is different from the tests for testing from
>    these types of JIDs.
>
>
> Given this thoroughness, the harness could approach 10,000 tests.  However,
> the vast majority of the tests are variations of sources, jid fields, and
> affiliations on a core set of about 200 tests.  These variations would be
> generated beforehand to a plan script that could be edited and overridden by
> hand for subtle exceptions.  Tests would be grouped into different
> categories like “owner tests” and “subscriber tests” and would be further
> broken down by individual features.  Test variations could be disabled by
> execution options for less thorough, quicker testing. Individual tests sets
> for a specific feature could also be invoked by the tester.
>
> Test plan scripts could be distributed to separate servers and instances of
> the test harness in order to provide load tests.  The execution of these
> plans would be coordinated from a single source.  Scaling tests could be
> done with a plan script that loops through certain tests (like node creation
> and subscription tests, following by node publishing and consuming tests),
> again potentially coordinated from multiple sources at the same time.
>
> The test harness would be developed in stages, so as to maximize it's
> usefulness before completion.  Between any of these steps, a user could
> still use the test harness, although obviously with only the previously
> developed features.  The development stages are as follows:
>
>    1. Develop core features tests (~200).
>    2. Add DTD compliance testing for expected namespaces.
>    3. Add detailed logging.
>    4. Create test plan generation and execution for test variations.
>    5. Create testing groups and features for managing which tests are
>    executed.
>    6. Add reporting for scoring and compliance levels.
>    7. Add web interface for executing tests.
>    8. Add test coordinator bot for scale and load testing.
>
> Obviously this is a very ambitious, very thorough test suite, and we could
make do without with hand-testing and smaller test suites for most XEPs.
This is simply an "ideal" test suite in my mind.  We should be making use of
those DTDs that you write, after all.

-Fritz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.jabber.org/pipermail/members/attachments/20090511/5782ecf2/attachment-0001.htm>


More information about the Members mailing list