[Standards-JIG] proto-JEP: Smart Presence Distribution

Carlo v. Loesch CvL at mail.symlynX.com
Thu Jun 1 14:48:43 UTC 2006

Richard Dobson typeth:
| How is it not a fact that your spec does not have a form of error 
| recovery that only needs to retransmit only what failed, rather than the 
| whole list to the server?

To the servers or the server? If you are talking about one single server,
then indeed, should the network be broken it will have to recreate the
list just for that single server. How many people do you know on jabber.org?
I have a pretty well distributed roster - people are a bit everywhere.
Additionally we can use Michals hashes, so we get to keep lists across
transport errors. Would be interesting to see how often or rare actual
transport errors happen.

Michal, how should we handle the hashes. Would you add them to the
presence smarticast, and do error recovery if they fail, or would you
burst out all hashes of all lists at connection linkup, no matter if
you actually need them. The first plan sounds saner to me.

| > you still haven't understood the proto-JEP properly.
| > no matter which part of the broad unicast fails and when, only that part
| > needs to be repeated or re-established. you don't have to restart
| > everything from scratch - you only need to redefine a couple of people
| > on the list of one particular host, if that particular host had serious
| > delivery problems.
| >   
| According to your spec, unless you have updated it very recently when 
| you get an error you reset the whole list, has this changed so that only 
| the failed items are resent? how did you accomplish this without acks?

No as I said the list is updated per host. I thought you were talking
about all hosts, equalling a complete new "broadcast."

| But it doesn't, you have to send the whole list (for a particular 
| server) again, rather than just the single JID at that server that you 
| were trying to add, you really do seem to be good at spinning facts and 
| what people are saying, you really should get yourself a job in politics 
| :P, hehe, but what I said is still true when related to the traffic 
| between two servers.

Ok I thought you were talking about *all* servers, because the traffic
that it takes to update one server's list doesn't really account for much.
And then you still have the hash.

| > | "To transmit a message to a select group of recipients. A simple example 
| > | of multicasting is sending an e-mail message to a mailing list."
| >
| > incorrect. why would one invent "ip multicast" if e-mail already did it.
| >   
| I see so you are saying *SES SIRIUS AB are liers then, hmm, interesting.

Well they have a strange notion of multicast, and unfortunately they are
not alone. But still what's the point in using the word for e-mail?
Why would you need the word for it then? If the word is reduced to the
meaning of sending something from here to more then one there. Than
one-to-many would be equal to multicast. Why say multicast if we already
say one-to-many. Why abuse the historic meaning of multicast?

| > this definition is mostly correct, although the intention is that every
| > packet gets sent to australia only once, so if you know people on 4
| > servers in australia, xmpp will, no matter which jep we apply to it,
| > send the packets 4 times towards australia. that's no longer proper
| > multicast then. it is what we call 'smarticast' at best.
| >   
| Just because all xmpp traffic for Australia doesn't get routed though a 

no, not all xmpp traffic, that would be rebuilding irc.
one smart router in australia for every context, chosen by the context.
a context is any multicast sender, like a groupchat or a person generating

| Australia, all the receiving servers in Australia could have an 
| agreement to share a single multi casting component for international 
| traffic, now when the server in France tries to discover the multi 
| casting component for the Australian domains they will all tell it to 
| use the same one, the server in France when it tries to communicate with 
| more than one of these Australian servers could realise this and 
| negotiate a single list with the multi casting component that covers all 

now we're talking.
that's the kind of thing we want to bring up in our follow-up
multicast jep that builds on top of this jep.
see, it is much more useful to use the word for nothing less than
the real thing, because it raises your aims to higher goals.  :)
if you let jep-0033 be multicast, no-one will ever sit down and
code the real thing.

| Also there is no such term as smarticast you are just making it up, 
| whereas an existing definition fits fine as you can see above.

we made it up because we had too much respect for multicast to
call this intermediate thing that way.

| My suggested list generation is not verbose when directly compared to 
| your method of sending lots of individual presence stanzas to build up 

ok, this one be valid.

| the list, and there are not lots of redundant acks, as I showed 
| demonstrated when initially setting up the list there is only the need 
| to send a single stanza which will get a single ack, to be classed as a 

yes, this is certainly better than acking each presence stanza, but
still you are sending an ack every time you make a change to the list,
whereas we know the list is healthy because our link is sane, so we
don't need *any* ack.

| bunch of acks if would have to be more than one. Also I've got no idea 

it is a bunch of acks because you have lists for every person on every
server you are talking to. so even your initial setup will get you acks
from all involved servers - tcp packets which are redundant because tcp
itself already ensures the lists are safe from harm.

| what you are going on about with regards to not needing to do 
| "broadcast" any more I thought that was the whole point of both of our 
| suggestions.

yeah ok it doesn't matter if it still looks like a traditional
presence fan out, or it looks like a fresh new <iq/> thing.

| Extra presence stanza??? Not sure what you mean by this either, could 
| you please explain.

after setting up iq you need an extra round trip for the ack, then you
can send the <presence to=router/>, which is also one more than in our

| But yes it does delay the initial presence delivery, but this will only 
| happen once as the lists will not need to be rebuilt and will be able to 
| be re-used whenever they reconnect, I find it very unlikely anyone would 

same goes for our lists, since you have to apply force to break a tcp
if both servers take care of closing links gently.

| if you are referring to the delay that will happen while waiting for the 
| S2S connections to establish between the domains then that's not really 

no, referring to the extra roundtrip delay for the <iq/>

| As already proved by the ack discussion TCP is not stable enough or that 
| discussion would not have happened in the first place, and there 
| wouldn't be any of the acking proposals, so again please stop repeating 
| your assertions that TCP is stable enough because there is plenty of 
| evidence to the contrary, you continuing to state this just demonstrates 
| a lack of full understanding of the issues at hand.

it is the dimensions in which you and i are thinking. i say occasional
breaks can be fixed, and it is cheaper than acking. you say breaks happen
all the time, so fixing costs more than acking. i say breaks no longer
happen all the time, because we enforce that servers stop closing idle
links irresponsibly. you say, that's not enough. i say, okay, so what
can we do now to actually find out who's right instead of keeping on

i have a suggestion. let's make it a negotation option to use acks.
with acks, the implementation can ignore tcp errors. without acks,
it will make good use of tcp stability. and the server administrator
gets to decide which strategy works best. or the implementation
switches acks on for servers that frequently burst into tears.

| If its true that on average there are only a few people on peoples 
| rosters then there is probably no point in even developing this 
| optimisation in the first place as it will mean it does not make any 
| real impact.

exactly, so let's keep on going.

| > if the amount of stanzas makes a difference though, than a single iq
| > with a lot of subtags is better than a series of presence stanzas.
| > please elaborate what the gains are moving away from the traditional
| > presence fan-out as it has always been, and on the other hand, if
| > anyone is contrary to the <iq> list setting strategy, please speak up now.
| >   
| Well using IQ means not just potential bandwidth savings when setting up 
| lists (especially large ones as the bandwidth savings will increase the 
| bigger the list), but it also means its easier for the server or 
| component to manage the list as it can all be done quickly in a single 
| operation rather than having to process lots of individual stanzas to 

sounds like an architectural issue here. would be useful to hear other
server developers. i know for our part it doesn't make a big difference
if we add list elements step by step or have it all at once. but in a
language like c one could even precalculate the size of memory the list
will take and allocate just the right amount. that does make sense.

| build up the list, also because XMPP stanzas are not guaranteed to be 
| delivered in the order they were sent (although on the whole they will 
| normally get there in order) having the built in acking of IQ means the 
| sending server knows once the operation has been completed and can then 

well that again can be solved differently. since the new presence is
sent to the router, the router will know that its list isnt completely
processed yet (oh how i love multithreading ;)) and wait until the
lists semaphore shows green. again for our implementation, such an
effect could not happen.

| start sending the presence stanzas to the component knowing that they 
| will definitely be delivered to the entire list that they set-up, 
| whereas with your spec there is a chance that if you quickly send a 
| presence stanza that is intended to be broadcast to the list along with 
| the set-up stanzas that the broadcast stanza could get processed before 
| all of the set-up stanzas have finished being processed.

no, because the presence messages being processed *are* the message.
and yes, because typically right after setting up the list, the probe
request comes along. okay so this would be an architectural issue for
server developers whose servers would not process the input of a single
tcp linearily. but, does this really happen? do you really process the
input of ONE SINGLE tcp stream in a non temporal fashion? do i really
have to take a scenario like this in consideration? how many other JEPs
would not be able to operate if things aren't happening in the right order?
i mean, do you get groupchat messages before the MUC has acked your entry?
aren't there a million scenarios that can't work if you're not keeping
the order of the stream?

[deleted the bit about connections and nats and etc, we have differing
opinions on that and i suggested a peaceful solution to that]

| connection will never be able to be restored) newer incoming connections 
| could be fine, but until the original connection times out you don't 
| know and all of the data sent to the socket in the mean time will be 
| completely lost.

if you get a new connection in, it is obvious that something went wrong
(or at least the other side thinks so), so you redo the list update.
so renewing a connection is okay.


| > it allows for
| > [x] unquoted and unparsed binary file transfers
| > [x] routing and multicasting content without losing time parsing it
| > [x] probably more
| My guess would probably be that you are alluding to SIP, but that's 

hm yeah i guess SIP qualifies too. i was obviously talking of PSYC.

| pretty much irrelevant, part of the whole point of XMPP is that it is a 
| streaming XML protocol, binary framing doesn't really fit in with this, 

XML was the hype thing in 1998, so it was used for IM.
it's not like a protocol was invented to do XML, then applied to IM.
but i know you didn't mean that. it just sounded that way.  :)

More information about the Standards mailing list