[Standards-JIG] proto-JEP: Smart Presence Distribution

[Richard Dobson]

> | Well it might be the end of the story for you but its still a big 
> | failing in your spec, it doesn't have a decent level of error recovery, 
>
> claim, no facts.
>   
How is it not a fact that your spec does not have a form of error 
recovery that only needs to retransmit only what failed, rather than the 
whole list to the server?
> you still haven't understood the proto-JEP properly.
> no matter which part of the broad unicast fails and when, only that part
> needs to be repeated or re-established. you don't have to restart
> everything from scratch - you only need to redefine a couple of people
> on the list of one particular host, if that particular host had serious
> delivery problems.
>   
According to your spec, unless you have updated it very recently when 
you get an error you reset the whole list, has this changed so that only 
the failed items are resent? how did you accomplish this without acks?
> | would be far better if your spec were able to only have to resend only 
> | those which did not get there.
>
> that's what it does. thanks for talking so much before reading it closely.
>   
But it doesn't, you have to send the whole list (for a particular 
server) again, rather than just the single JID at that server that you 
were trying to add, you really do seem to be good at spinning facts and 
what people are saying, you really should get yourself a job in politics 
:P, hehe, but what I said is still true when related to the traffic 
between two servers.
> | "Like a broadcast, a multicast replicates information from one source to 
> | multiple recipients on a network"
>
> incorrect.
>   
I see, so you are saying google and all the sites it is indexing for the 
definitions are incorrect, hmmm, I'm sorry but I'm far more inclined to 
believe these google results than you.
> | "To transmit a message to a select group of recipients. A simple example 
> | of multicasting is sending an e-mail message to a mailing list."
>
> incorrect. why would one invent "ip multicast" if e-mail already did it.
>   
I see so you are saying *SES SIRIUS AB are liers then, hmm, interesting.
*
> | "Special form of broadcast in which copies of the packet are delivered 
> | to multiple stations, but only a subset of all possible destinations"
>
> just go into the IETF MMUSIC WG and ask them if these are valid
> definitions of "multicast" - they will, if they spare the time, inform
> you that these are all superficial simplifications of what multicast
> actually does.
>   
Even if they are simplified they are still valid definitions, can we 
just put an end to this, you are clearly mistaken as far as I can see 
and I doubt you will ever be able to convince me that your definition is 
the only possible valid one, so there really is no point continuing this 
part of the discussion.
> | "Multicast is the delivery of information to a group of destinations 
> | simultaneously using the most efficient strategy to deliver the messages 
> | over each link of the network only once and only create copies when the 
> | links to the destinations split. By comparison with multicast, 
> | conventional point-to-single-point delivery is called unicast, whereas 
> | delivery to every node of the network is broadcast"
>
> this definition is mostly correct, although the intention is that every
> packet gets sent to australia only once, so if you know people on 4
> servers in australia, xmpp will, no matter which jep we apply to it,
> send the packets 4 times towards australia. that's no longer proper
> multicast then. it is what we call 'smarticast' at best.
>   
Just because all xmpp traffic for Australia doesn't get routed though a 
single point and then fanned out from there doesn't in and of itself 
mean it can't be described as multi casting as its not the countries 
that are interested in this traffic, its individual server domains, but 
even so my suggested spec could operate in the way you suggest above if 
necessary, the multi casting component could possibly be shared by 
multiple servers when using my suggestion, when the external servers are 
discovering the multi casting component to use for a particular domain 
say the sending domain is in France and the receiving server is in 
Australia, all the receiving servers in Australia could have an 
agreement to share a single multi casting component for international 
traffic, now when the server in France tries to discover the multi 
casting component for the Australian domains they will all tell it to 
use the same one, the server in France when it tries to communicate with 
more than one of these Australian servers could realise this and 
negotiate a single list with the multi casting component that covers all 
the Australian servers that share this multi casting component, easy, 
there could also be regional multi casting components that handle 
subsets of these servers and there is no real reason that the multi 
casting components couldn't negotiate lists with each other to further 
fan out the data (although this is getting a little over complicated), 
so you see even by your too narrowly defined definition of multicast, my 
suggestion could certainly be described as multicast as it is physically 
capable of doing it (although whether people would want to set it up 
that way is another question entirely due to all the trust and potential 
spam issues that arise), so please just drop your assertions about us 
being wrong to use the term multicast, as even by your incorrectly 
narrowly defined definition its fine, even JEP-0033 could work in a tree 
distribution style if you really tried.

Also there is no such term as smarticast you are just making it up, 
whereas an existing definition fits fine as you can see above.
> | Protocol verbosity??? I'm confused, how does reducing the number of 
> | stanzas significantly and easily providing for error recovery mean its 
> | verbose, if anything its less so due to the major reduction in the 
>
> your list generation is verbose, maybe not much more than a "broad unicast"
> of presence though, so if the good ole "broadcast" is no longer wanted, it
> might as well be an iq. we still get to send an extra <presence/> stanza
> to all links, and we have a bunch of redundant acks, since TCP is stable
> enough for the purpose. so all in all your proposal is still more verbose
> and additionally it requires an extra roundtrip for the acks which delays
> presence delivery, as dave pointed out.
>   
My suggested list generation is not verbose when directly compared to 
your method of sending lots of individual presence stanzas to build up 
the list, and there are not lots of redundant acks, as I showed 
demonstrated when initially setting up the list there is only the need 
to send a single stanza which will get a single ack, to be classed as a 
bunch of acks if would have to be more than one. Also I've got no idea 
what you are going on about with regards to not needing to do 
"broadcast" any more I thought that was the whole point of both of our 
suggestions.

Extra presence stanza??? Not sure what you mean by this either, could 
you please explain.

But yes it does delay the initial presence delivery, but this will only 
happen once as the lists will not need to be rebuilt and will be able to 
be re-used whenever they reconnect, I find it very unlikely anyone would 
really notice this, its unlikely to take more than a second or two to 
set-up the list due to the fact it can all be done in a single stanza, 
if you are referring to the delay that will happen while waiting for the 
S2S connections to establish between the domains then that's not really 
an issue either as until that is established you wouldn't be able to be 
communicating with the people at that server anyway or have even 
received their presence in the first place to know that they are 
on-line, so again not an issue.

As already proved by the ack discussion TCP is not stable enough or that 
discussion would not have happened in the first place, and there 
wouldn't be any of the acking proposals, so again please stop repeating 
your assertions that TCP is stable enough because there is plenty of 
evidence to the contrary, you continuing to state this just demonstrates 
a lack of full understanding of the issues at hand.
> 50 stanzas? didn't stpeter once claim average users have 10 people in their
> rosters? so why should they have 50 people all on one server? anyway,
> amount of people doesn't matter.
>   
If its true that on average there are only a few people on peoples 
rosters then there is probably no point in even developing this 
optimisation in the first place as it will mean it does not make any 
real impact.
> if the amount of stanzas makes a difference though, than a single iq
> with a lot of subtags is better than a series of presence stanzas.
> please elaborate what the gains are moving away from the traditional
> presence fan-out as it has always been, and on the other hand, if
> anyone is contrary to the <iq> list setting strategy, please speak up now.
>   
Well using IQ means not just potential bandwidth savings when setting up 
lists (especially large ones as the bandwidth savings will increase the 
bigger the list), but it also means its easier for the server or 
component to manage the list as it can all be done quickly in a single 
operation rather than having to process lots of individual stanzas to 
build up the list, also because XMPP stanzas are not guaranteed to be 
delivered in the order they were sent (although on the whole they will 
normally get there in order) having the built in acking of IQ means the 
sending server knows once the operation has been completed and can then 
start sending the presence stanzas to the component knowing that they 
will definitely be delivered to the entire list that they set-up, 
whereas with your spec there is a chance that if you quickly send a 
presence stanza that is intended to be broadcast to the list along with 
the set-up stanzas that the broadcast stanza could get processed before 
all of the set-up stanzas have finished being processed.
> yes i understand the amazing manifold world of timeout problematica,
> but still the most common reason for roster asyncs could be as simple
> as server downtimes. luckily our proto-jep doesn't need to do anything
> if the other side isn't available, so it can always limit its operation
> to a healthy working connection between two entities.
>   
You still seem to be missing the point, its not just server downtime 
that causes these connection time-out problems, server downtime will 
most likely cause the connections not to be able to be established in 
the first place, the main cause of these time-out problems is 
intermittent problems with things like firewalls/routers/NATs where 
already established connections become broken (which is not able to be 
detected often for some time), whereas even while the connection is 
broken (which because of things like NATs will likely mean the 
connection will never be able to be restored) newer incoming connections 
could be fine, but until the original connection times out you don't 
know and all of the data sent to the socket in the mean time will be 
completely lost.
> it allows for
> [x] unquoted and unparsed binary file transfers
> [x] routing and multicasting content without losing time parsing it
> [x] probably more
>
> even if we get xmpp to do real multicast, it's a bit sad we cannot
> use it for transparent transfers. you probably don't wanna know
> which technology is suited for all of those [x] up there.
>   
My guess would probably be that you are alluding to SIP, but that's 
pretty much irrelevant, part of the whole point of XMPP is that it is a 
streaming XML protocol, binary framing doesn't really fit in with this, 
but all it means is that everything sent over the socket must be valid 
XML so its certainly possible to use it for transfers, it just means 
they need to be wrapped up in an XML compatible way which is not exactly 
very difficult, have a look at JEP-0047.
> if the jep makes a precise definition on the sort of change it MUST
> make in order for the distribution system to operate correctly, that
> is a very very unfree change of an attribute, and not of the content.
>   
Better to avoid if at all possible, although I've got a feeling that it 
will probably be the only way to do it that will work in-line with the 
rest of your proposal as it stands.
> also we still haven't dismissed the option that we could as well
> use the recipients roster, since the scenario of somebody hacking his
> roster when he also needs to receive directed presence *and* needs to
> be on the same server with more recipients, makes it a rather faint
> security weakness. it's easier to talk somebody into relaying you
> somebody else's presence.
>   
Well I certainly thought we had already dismissed it, but if you really 
want to take a step back to that again then feel free, its up to you, 
but I very much doubt you will be any more successful than you were last 
time around.
> so if we use the second scenario, we don't even need to modify directed
> presence.
>   
The roster scenario you mean? Well that's up to you but I find it 
unlikely many people are going to be prepared to implement that solution 
given all the problems that have already been pointed out to you.
> that is your presumption, but by all logic i can't see how a lot of
> tcp connections should keep on breaking continously when both ends
> could have closed them properly.
>   
Well then it sounds like you possibly don't have enough real world 
experience of TCP/IP networking.
> but sure, we could add an extra clause that stops presence distribution
> on unreliable links. i mean, if a network connection isn't reliable, then
> it is experiencing some serious problems. then i would want important
> protocols like ssh to get all the priority. the least i need on a messed
> up network is some people's away messages.
>   
Well I would suggest you add the acking protocol cause in right away 
then, and wait for it to be widely implemented.

Richard