[Members] First XSF Certificate Meeting

Peter Saint-Andre stpeter at stpeter.im
Mon May 11 17:56:44 CDT 2009


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 5/11/09 4:37 PM, Nathan Fritz wrote:
> 
>     "Certificate" makes me think of X.509. Perhaps we could call it
>     "Certification" instead.
> 
> 
> What's in a name?  Sure, certification is fine.
> 
>     Another action item would be to research open-source testing tools.
>     Perhaps http://www.opensourcetesting.org/ has some useful links.
> 
>     It seems to me that we'd want to develop an extensible approach that
>     enables XSF members and interested others to write their own test sets
>     for XMPP functionality. Something scriptable comes to mind. :)
> 
> 
> Sure, I think that expecting everyone to contribute to a single harness
> in a single language would be ideal but is a fantasy.  I'll look into
> opensourcetesting.org <http://opensourcetesting.org>.  Perhaps expecting
> specific result from a test harness would allow us to span all of the
> languages that our members enjoy, but still us to have a unified interface.

It strikes me that developing something in [insert-language-here] and
using that across the board is less of a fantasy than trying to use the
same tests with multiple test harnesses in multiple languages.

Every serious XMPP project / company I've known has their own testing
systems. It's not clear to me if the test cases are tied to the language
in which each harness is written.

> For example, our harness could run as a webapp, and call individually
> contributed tests (in various languages) calling each with a
> standardized configuration XML format, and returning a standardized XML
> result so that the webapp could display and total the results from
> disparate tests in a unified way.  This seems like a "down the road"
> idea, and I think we'd be better served by starting without at first. 
> We could work up to it.

Possible. But I think we'd need to bootstrap it in a specific language
first -- Python, Ruby, C#, Java, Lua, whatever.

>     What's the general approach behind your test harness?
> 
> 
> Here's something I wrote to explain it to someone else:
> 
>     XMPP Publish-Subscribe Test Harness
> 
>     This test harness is designed to assist QA staff in finding bugs,
>     provide developers with clear goals and tasks to develop against,
>     and build customer confidence in specification compliance and
>     stability.  The harness is available via command line or website, is
>     cross platform, can email detailed results, and be automated with a
>     check-in procedure.  A configuration file determines credentials for
>     JIDs to test from, values to test against (for example, a size of
>     pubsub payload that should cause an “payload too large error”) and
>     other values.  The test will be scored based on various compliance
>     levels, as well as a raw pass/fail ratios.  This harness is designed
>     to be applied to other areas of XMPP compliance testing, but is not
>     a generic test suite for any other wire protocol.
> 
>     The following tests will be applied to *each feature* in XEP-0060:
> 
>         * Tests that check for proper error responses.
>         * DTD compliance tests for all incoming and outgoing traffic
>           during each test.
>         * Tests from local accounts, federated accounts, and
>           SASL-Anonymous bound JIDs
>         * Tests for each node affiliation against every applicable feature.
>         * Tests that compare bareJIDs, fullJIDs, and federated, local,
>           and anonymous JIDs in every combination reasonable against
>           every JID field for every applicable feature. This is
>           different from the tests for testing from these types of JIDs.
> 
> 
>     Given this thoroughness, the harness could approach 10,000 tests. 
>     However, the vast majority of the tests are variations of sources,
>     jid fields, and affiliations on a core set of about 200 tests. 
>     These variations would be generated beforehand to a plan script that
>     could be edited and overridden by hand for subtle exceptions.  Tests
>     would be grouped into different categories like “owner tests” and
>     “subscriber tests” and would be further broken down by individual
>     features.  Test variations could be disabled by execution options
>     for less thorough, quicker testing. Individual tests sets for a
>     specific feature could also be invoked by the tester.

What is the difference between "test set" and "test script"? What are
these written in?

>     Test plan scripts could be distributed to separate servers and
>     instances of the test harness in order to provide load tests.  The
>     execution of these plans would be coordinated from a single source. 
>     Scaling tests could be done with a plan script that loops through
>     certain tests (like node creation and subscription tests, following
>     by node publishing and consuming tests), again potentially
>     coordinated from multiple sources at the same time.
> 
>     The test harness would be developed in stages, so as to maximize
>     it's usefulness before completion.  Between any of these steps, a
>     user could still use the test harness, although obviously with only
>     the previously developed features.  The development stages are as
>     follows:
> 
>        1. Develop core features tests (~200).
>        2. Add DTD compliance testing for expected namespaces.
>        3. Add detailed logging.
>        4. Create test plan generation and execution for test variations.
>        5. Create testing groups and features for managing which tests
>           are executed.
>        6. Add reporting for scoring and compliance levels.
>        7. Add web interface for executing tests.
>        8. Add test coordinator bot for scale and load testing.
> 
> Obviously this is a very ambitious, very thorough test suite, and we
> could make do without with hand-testing and smaller test suites for most
> XEPs.  This is simply an "ideal" test suite in my mind.  

Yes, that's quite comprehensive! I like the idea of starting small
(immediate bang for the buck) and then we can build outward from there.
It'd be really cool to have some of this in place before we work to
advance the XMPP RFCs from Proposed Standard to Draft Standard (probably
in 2010 or 2011)), because the IETF requires interop reports. We'd blow
their doors off with complete testing results like this. :)

> We should be
> making use of those DTDs that you write, after all.

Or W3 XML Schema, or Relax NG -- whatever is easiest to use!

Peter

Peter

- --
Peter Saint-Andre
https://stpeter.im/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkoIrSwACgkQNL8k5A2w/vxLvwCdELmKSR3lRXFskV78HGBF5PGJ
TbUAnR4DKeONkdmj2fYbdwP5eY3gxyhs
=ZSii
-----END PGP SIGNATURE-----


More information about the Members mailing list