[Standards-JIG] In order delivery for xep-0047 ?

JD Conley jd.conley at coversant.net
Fri Dec 15 11:13:50 CST 2006


Mridul,

Usually we're on the same page in these debates, but I think we're on different planets for this one. :) I can certainly see how if one interpreted the spec to not mean in order delivery it could lead to all sorts of design assumptions, but I also believe in order delivery is the intent of the spec.

> Initial stanza's from C would get queue'ed (which connection is in
> progress) and it is possible that a later set of stanzas (from C) could
> get delivered before as soon as the connection is established (not
> really a bug but a design compromise, since fixing this normally would
> be expensive).

This just sounds like a broken implementation to me. Why doesn't everything go in the queue? There are a ton of race conditions around connections where not having a _stanza level_ queue will cause out of order delivery...

> I do agree that I am describing a slightly contrived (though common
> enough) usecase here, but just to illustrate that out of order delivery
> should not really be so surprising.

Out of order delivery is _very_ suprising to some of us and, IMO, should not be allowed except in well defined edge cases (see below).

> Actually, there are a bunch of corner cases where xmpp leads to race
> conditions (rosters and privacy lists comes to mind). Some of which are
> handled optimistically by implementations, and some others are
> mentioned
> here :-)

Definitely won't argue with this, but it's a whole other thread. 

> > You are right, that in-order delivery is an additional burden for
> server
> > implementations, but it is not something that prevents scaleablility,
> it
> > just requires additional work by the implementor.
> 
> In the above case for example, you can imagine the amount of critical
> sections/locking to enforce in order delivery - and this is going to
> affect all stanza's to be sent from S1 to S2 - always : If nothing
> else,
> that would kill any server.

Burden? Compared with the presence engine and MUC and Pubsub, I don’t think so. Kill? That's a bit harsh... it's just a software implementation detail that needs a well thought out design. Yes it's complicated, but that's why we get the big bucks. :)

> Another simple example where in order delivery would actually be very
> detrimental for xmpp-system performance would be IBB itself.
> That is, in the above example which you described C -- S1 -- S2
> If you allow all traffic from C to S2 on same set of sockets - you will
> quickly end up saturating the S1 -- S2 link with C's ibb stanza's : not
> just for C, and this affects all clients hosted on S1.
> Alternative is to use another ibb specific s2s out from S1 to S2 ...
> and
> use that for ibb stanzas for clients on S1.
> Please do note that, in the case above we are totally breaking in order
> delivery - though from ibb's channels point of view, we are not.

Again, IIRC, the intent of the spec was in order delivery. In any case though, I think an exception can be made for IBB (and maybe some other edge cases), which would be very handy to be able to put in a priority queue, giving more time sensitive data the go-ahead. However, I will argue to my grave that the queue of IBB stanzas themselves MUST be delivered in order. Also, any other stanzas MUST NOT be delivered out of order.

> The burst traffic and stanza size in IBB is typically 'large', order of
> magnitude larger than a normal xmpp : and there is absolutely no way to
> even specify any form of flow control - other than relying on external
> means of traffic shaping. In our server, we special case IBB with a
> bunch of code just to ensure in order delivery and make sure that
> clients dont break cos of the strict in-order requirement : and this
> comes at a non-trivial performance penalty.

The IQ version of IBB was created for the purpose of flow control.

I still think IBB should be in order. If you want to get into performance penalties... How about the fact that we have to parse an entire XML stanza rather than just the addressing portions to be able to deliver it. XMPP design isn't catered to raw performance. It's about an easy to understand, powerful, flexible, real time messaging and presence protocol. Heck, why do we use XML at all? It's a CPU hog! Any time we do any sort of performance tuning in our implementations XML serialization is by far the #1 offender. Let's just ditch that too! How about a nice binary protocol I can just load directly into a struct? Ok, maybe I took that a bit far. :)

> 
> >  Remember, that we do
> > not only have clients running on PCs with lots of memory and
> computing
> > power. We also have clients running on mobile devices with maybe just
> a
> > bit of RAM and no access to other storage.
> >
> 
> The problem I describe above is not something which happens commonly :
> but happens under load (when you have quiet a bit of data queued up for
> a client which starts to cause socket buffer to be full).

Again, this really seems like an implementation issue. Every proxying system on the planet has this issue. Which really, in the IBB case, we really are acting as a proxy.

> A potential solution would be for the client to have a simple
> implementation/config dependent window based timeout for ibb stanza's.
> Since all stanza's are  identified with a seq-id - the window size can
> be as large as 64k/2 or as small as 1 (that is not handle it - current
> case). But as long as spec has those MUST clauses, clients or
> applications cannot support this without being non-compliant, and
> servers have to resort to tricks to handle it.

This is one of the main reasons for the addition of IQ to the spec. The client waits for that result, which the server could certainly queue up for rate limiting purposes, before sending the next set of data.

> For a normal use case of chat client, the possibility of out of order
> delivery is negligible : typically the traffic load is minimal.
> If you are having applications which use xmpp as middleware and are
> going to pump messages at a high rate , then it is quite simple to have
> some sort of stanza id'ing to handle potential out of order delivery.

The possibility of out of order delivery MUST be 0. If I'm firing off 1000 messages/second in a stock trading application and they need to be processed and displayed instantly on a real time graph, they simply can't be out of order. If that were a possibility we might as well base our entire protocol on UDP (I think someone mentioned this earlier). It's much more performant in the general case and, if you don't have high throughput, will usually deliver whatever you want without any sort of rate throttling!

In general we all try to avoid IBB unless it's strictly necessary. That's why we have bytestreams and jingle! Though in the proxied case they even suffer from the same sorts of buffering issues. Heck, why should we even delivery those binary packets in order?! It limits server scalability!.

We really need to keep things in order in each of our implementations as much as possible, unless it is certain that delivering them out of order won't cause any confusion. Giving priority to data other than IBB in a stream for QoS purposes sounds like a great idea. Re-ordering the IBB stream itself does not.

-JD


More information about the Standards-JIG mailing list