[Standards-JIG] In order delivery for xep-0047 ?
mridul at sun.com
Fri Dec 15 23:21:28 UTC 2006
Maybe I am not explaining it properly enough.
Essentially any xmpp server always tries to make best case effort to
deliver in order.
But it is just a best case effort : if we enforce MUST for in order
delivery, then even in worst case scenario, it must still deliver in order.
Trying to make the worst case scenario compliant is what causes problems
: and ibb is one case were you hit worse case very fast as you fill up
socket and application level buffers while clients transfer large enough
files (unless server starts throttling client or uses external means to
do so : ignore those for time being please - impl details, not protocol
I am thinking of case where server tries to handle IO asynchronously.
Hopefully the extreme cases I describe below will be viewed under these
JD Conley wrote:
> Usually we're on the same page in these debates, but I think we're on different planets for this one. :) I can certainly see how if one interpreted the spec to not mean in order delivery it could lead to all sorts of design assumptions, but I also believe in order delivery is the intent of the spec.
>> Initial stanza's from C would get queue'ed (which connection is in
>> progress) and it is possible that a later set of stanzas (from C) could
>> get delivered before as soon as the connection is established (not
>> really a bug but a design compromise, since fixing this normally would
>> be expensive).
> This just sounds like a broken implementation to me. Why doesn't everything go in the queue? There are a ton of race conditions around connections where not having a _stanza level_ queue will cause out of order delivery...
To reduce contention - under normal case (when outbound stream is open),
you just write the stanza and thread is done with it.
You wont need another thread to push and pop from a queue : btw, that is
a model that is followed by some implementations (not jsut for xmpp),
but it has not worked well for us till now.
>> I do agree that I am describing a slightly contrived (though common
>> enough) usecase here, but just to illustrate that out of order delivery
>> should not really be so surprising.
> Out of order delivery is _very_ suprising to some of us and, IMO, should not be allowed except in well defined edge cases (see below).
>> Actually, there are a bunch of corner cases where xmpp leads to race
>> conditions (rosters and privacy lists comes to mind). Some of which are
>> handled optimistically by implementations, and some others are
>> here :-)
> Definitely won't argue with this, but it's a whole other thread.
>>> You are right, that in-order delivery is an additional burden for
>>> implementations, but it is not something that prevents scaleablility,
>>> just requires additional work by the implementor.
>> In the above case for example, you can imagine the amount of critical
>> sections/locking to enforce in order delivery - and this is going to
>> affect all stanza's to be sent from S1 to S2 - always : If nothing
>> that would kill any server.
> Burden? Compared with the presence engine and MUC and Pubsub, I don’t think so.
Presence, muc and pubsub typically do not involve IO delays - you do
have processing overhead for sure.
Enforcing in order delivery could incur IO delays in server processing
thread in worst case scenarios.
> Kill? That's a bit harsh... it's just a software implementation detail that needs a well thought out design. Yes it's complicated, but that's why we get the big bucks. :)
Heh, true :-)
Just to mention, we do handle ibb as of now by special casing it - and
other than while transferring pretty large files between clients over
ibb, we do not see out of order delivery. That being said, it would be
preferable for ibb itself to handle stanza ordering. ibb spec makes the
design assumptions it does based on gaurenteed in-order delivery :
something which is a bit tough to always ensure.
>> Another simple example where in order delivery would actually be very
>> detrimental for xmpp-system performance would be IBB itself.
>> That is, in the above example which you described C -- S1 -- S2
>> If you allow all traffic from C to S2 on same set of sockets - you will
>> quickly end up saturating the S1 -- S2 link with C's ibb stanza's : not
>> just for C, and this affects all clients hosted on S1.
>> Alternative is to use another ibb specific s2s out from S1 to S2 ...
>> use that for ibb stanzas for clients on S1.
>> Please do note that, in the case above we are totally breaking in order
>> delivery - though from ibb's channels point of view, we are not.
> Again, IIRC, the intent of the spec was in order delivery. In any case though, I think an exception can be made for IBB (and maybe some other edge cases), which would be very handy to be able to put in a priority queue, giving more time sensitive data the go-ahead. However, I will argue to my grave that the queue of IBB stanzas themselves MUST be delivered in order. Also, any other stanzas MUST NOT be delivered out of order.
I mentioned the above to illustrate one of the ways we had tried to
reduce the load of multiple ibb transfers and not cause it to affect
'normal' s2s and server pool traffic. ibb causes more of a problem for
us since we support file transfer to a conference over ibb - you can
imagine the traffic going over the tcp stack in that case wen a
reasonably large file is multicasted.
>> The burst traffic and stanza size in IBB is typically 'large', order of
>> magnitude larger than a normal xmpp : and there is absolutely no way to
>> even specify any form of flow control - other than relying on external
>> means of traffic shaping. In our server, we special case IBB with a
>> bunch of code just to ensure in order delivery and make sure that
>> clients dont break cos of the strict in-order requirement : and this
>> comes at a non-trivial performance penalty.
> The IQ version of IBB was created for the purpose of flow control.
I think it is not mandatory to wait for response ... but you are right,
atleast there is some way for clients to minimize mad push of stanza's
using the iq version.
> I still think IBB should be in order. If you want to get into performance penalties... How about the fact that we have to parse an entire XML stanza rather than just the addressing portions to be able to deliver it. XMPP design isn't catered to raw performance. It's about an easy to understand, powerful, flexible, real time messaging and presence protocol. Heck, why do we use XML at all? It's a CPU hog! Any time we do any sort of performance tuning in our implementations XML serialization is by far the #1 offender. Let's just ditch that too! How about a nice binary protocol I can just load directly into a struct? Ok, maybe I took that a bit far. :)
(un ?) fortunately, point is to spread the pain to few server so that
much larger set of client and end users of xmpp can benifit, but still I
totally agree :-)
>>> Remember, that we do
>>> not only have clients running on PCs with lots of memory and
>>> power. We also have clients running on mobile devices with maybe just
>>> bit of RAM and no access to other storage.
>> The problem I describe above is not something which happens commonly :
>> but happens under load (when you have quiet a bit of data queued up for
>> a client which starts to cause socket buffer to be full).
> Again, this really seems like an implementation issue. Every proxying system on the planet has this issue. Which really, in the IBB case, we really are acting as a proxy.
The difference being that a file transfer or proxying service is
designed to handle large traffic - while an xmpp system is designed to
handle smaller chunks from a much larger concurrent set of users with
additional requirements on how it gets processed. We end up designing
the impl based on the assumptions : I would definitely do the server
differently if main intent was to perform better in file transfer case,
and I dont think we would support as many concurrent user sessions as we
>> A potential solution would be for the client to have a simple
>> implementation/config dependent window based timeout for ibb stanza's.
>> Since all stanza's are identified with a seq-id - the window size can
>> be as large as 64k/2 or as small as 1 (that is not handle it - current
>> case). But as long as spec has those MUST clauses, clients or
>> applications cannot support this without being non-compliant, and
>> servers have to resort to tricks to handle it.
> This is one of the main reasons for the addition of IQ to the spec. The client waits for that result, which the server could certainly queue up for rate limiting purposes, before sending the next set of data.
I think the spec says that the client need not wait for the response ...
but it could, and that would help.
>> For a normal use case of chat client, the possibility of out of order
>> delivery is negligible : typically the traffic load is minimal.
>> If you are having applications which use xmpp as middleware and are
>> going to pump messages at a high rate , then it is quite simple to have
>> some sort of stanza id'ing to handle potential out of order delivery.
> The possibility of out of order delivery MUST be 0. If I'm firing off 1000 messages/second in a stock trading application and they need to be processed and displayed instantly on a real time graph, they simply can't be out of order. If that were a possibility we might as well base our entire protocol on UDP (I think someone mentioned this earlier). It's much more performant in the general case and, if you don't have high throughput, will usually deliver whatever you want without any sort of rate throttling!
> In general we all try to avoid IBB unless it's strictly necessary.
Unfortunately, there are a lot of cases where ibb becomes very useful -
when both (or single) client is behind firewall, connected through
proxy/124 gateway, multicasting to a conference, etc.
> That's why we have bytestreams and jingle! Though in the proxied case they even suffer from the same sorts of buffering issues. Heck, why should we even delivery those binary packets in order?! It limits server scalability!.
Reading and writing content from a pair of sockets can be handled
trivially by a very small set of threads for a very large number of
sockets - not sure what the problem there would be.
> We really need to keep things in order in each of our implementations as much as possible, unless it is certain that delivering them out of order won't cause any confusion. Giving priority to data other than IBB in a stream for QoS purposes sounds like a great idea. Re-ordering the IBB stream itself does not.
As a protocol, xmpp assumes exchange of small stanza's ...
Which was why I was special casing ibb in the earlier illustration to
show how we special case it since it is a bending this assumption.
Btw, I am currently working on improving and making ibb scale - which is
why all worst case scenarios and usecases seem to be so common to me :-)
More information about the Standards