[Members] interop testing

[Dave Cridland]

On Sat Oct 30 14:17:05 2010, Peter Saint-Andre wrote:
> I think it would be good to start preparing better for interop  
> testing
> at XSF events. I'm fine at leading discussions about specs, but I'm  
> not
> a developer so I don't know quite what we need here. What do you  
> folks
> think? How can the XSF help in terms of process and infrastructure?

On this note...

I'd like to propose that the XSF arranges regular interop events to  
test and publicly demonstrate interoperability across a range of  
functionality.

Various other organizations are doing the same, at the moment, and  
these are only involving a small cross-section of the implementations  
- generally those focussed on a particular industry sector. This  
might cause a somewhat skewed viewpoint on just how good  
interoperability *is* in XMPP - I've always assumed it to be  
excellent, but involvement with some of these specialist events has  
shaken my confidence, and not done XMPP itself any favours.

Therefore, I'm proposing that the XSF should organize an event prior  
to this; this suggests rather strongly that the event should be  
conducted online. The things I'd like to concentrate on are the  
basics - S2S and C2S authentication/security, and MUC. (Yes, this  
does imply existing implementations failed to interop even here, much  
to my surprise).

In particular:

- TLS/X.509 authentication for S2S.
- SCRAM (including, ideally, -PLUS) for C2S.
- Joining local and remote chatrooms.

I also think we could do more, but I'd ideally like to conentrate on  
just this. Other technologies seem to be easier to do interop testing  
in somewhat less formal ways, in part because the implementation  
pools are often smaller.

I'm suggesting the first week of December - that is, Monday 6th  
through to Saturday 11th, inclusive.

The way I see it operating is as follows:

1) The XSF sets up a test DNS zone, containing domains for each  
implementation taking part (this might well be more than one - in  
Isode's case, I think we'd be likely to put in our current release as  
well as the forthcoming one).

2) Participants can arrange times to perform testing. Participants  
will provide each other with transcripts as needed. Server  
implementations would be online continuously for the week; the  
arrangement of times is for the "staff".

3) Two temporary teams need to be formed by the XSF. One  
administrative, to organize resources such as the DNS, chatrooms,  
mailing lists, etc. One technical, for arbitration if we actually get  
an interop failure (and writing up what the correct reading and/or  
fix the specifications need).

4) The XSF will need to operate a CA for the event. Whilst I've not  
confirmed this, I think Isode can provide (at least one) copy of our  
CA software to the XSF for this purpose, which would give somebody a  
simple (ie, GUI clicky) way of generating X.509 certificates with  
various combinations of Subject RDNs and Subject Alternative Names.

So, questions:

a) Are sufficient numbers of members interested in this?

b) Are sufficient number of implementors able to commit? (I know many  
aren't members, but if we can generate a critical mass here, we  
should be fine in attracting more).

c) Is the Board (presumably) interested in overseeing the  
organization?

Dave.
-- 
Dave Cridland - mailto:dave at cridland.net - xmpp:dwd at dave.cridland.net
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade