[Standards] Council on Stanza Repeaters without Multicast

[Pedro Melo]

On Apr 3, 2008, at 9:48 AM, Dave Cridland wrote:
> On Thu Apr  3 08:23:36 2008, Pedro Melo wrote:
>>> A) Compression
>>> It's crucial on two counts:
>>> 1) It's much simpler to implement, and2) Given that we are (or   
>>> should be) encrypting every S2S connection, then TLS is giving  
>>> us  compression anyway, and moreover, it's cheaper to compress  
>>> than not  to compress.
>>> We've been down the road before where on some admittedly fairly   
>>> simple figures I did a while back - and repeated for Peter's   
>>> scalability draft in the SIMPLE WG, I think - the existing  
>>> presence  "splurges" compress really well, often better than the  
>>> "optimized"  alternatives.
>>> Smart Presence was one proposal that beat pure compression, as I   
>>> recall, but it has security implications which I'm not going to   
>>> accept.
>> Bandwidth is not the only concern. If I have a very popular  
>> pubsub  topic (like a Atom node for some high-profile blog or  
>> social site)  with hundreds of thousands of subscribers, the cost  
>> of expanding in  the source is very high, in terms of CPU also.
> Well, yes and no - XML parsing appears to hit a lot of servers  
> hard, but routing is routing, and repeaters will increase the cost  
> there, I suspect. If you're having to expand routers, that  
> expansion has to happen somewhere - it's a database lookup, or a  
> filesystem read, or a hash table in memory - but it's certainly  
> additional activity.
>
> This cost might, in some implementations, be outweighed by the  
> saving in XML parsing. I don't know, and I don't pretend to know,  
> either. I'm pretty sure it isn't in our implementation.

If you ignore list changes for a minute, routing costs will be the  
same on remote nodes (the same amount of stanzas will be routed + 1,  
actually) with or without repeaters. The biggest savings are in the  
source service + source router which will have much less stanzas  
transmitted and routed.

So globally, if you look at the cost of delivering a single  
notification to thousands of subscribers, the cost to the XMPP  
network as whole is lower with repeaters, and given the the fan-out  
processing is pushed to the edges (and therefore the costs of lookup  
are distributed amongst interested parties), I would be willing to  
bet that the latency from source to destination will also be lower, a  
small bonus.

So I do not agree that repeaters will increase the routing cost here.  
Maybe a little on the remote hosts, but will save a lot on the source  
router.

As for XML processing, well, if/when that becomes a problem, we can  
also use a xep-0138 profile for some more efficient representation  
format for XMPP stanzas. For example, two ejabberd servers could use  
native erlang message payloads when talking to each other... All is  
fair you pre-declare.


>> The mailing list people moved to sub-lists to deal with this.   
>> Repeaters are just like sub-lists. Please note that I'm talking  
>> of  the *public* pubsub node use-case.
> Right, almost. Repeaters are like sub-lists controlled by the  
> sender - that's a quite strinking difference.

No, repeaters are sub-lists controlled by the service/the list. If I,  
the sender, publish a node to some pub-sub service, I don't know nor  
I even care if the pubsub service is using repeaters.

The sender has no control on the repeaters.


>>> B) RTT delays
>>> The introduction of Stanza Repeaters gives us RTT delays whenever  
>>> a  list needs changing. That just strikes me as worrying - I  
>>> don't  think anyone has clear figures on how much delay would be   
>>> introduced, but I'm sure you'll agree that additional latency  
>>> does  not a performance improvement make.
>>> It may be possible to alter the protocol to remove these.
>> This is important if and only if list changes are the most  
>> frequent  operation.
>> If sending messages through the repeaters is the most common   
>> operation, then the lower RTT of distribution that Stanza  
>> Repeaters  give us, will over-shadow the delays with list changing.
> Right, kind of - for any given repeater, you need a list-change per  
> event to make it happen every time. And I'm not sure where a  
> suitable breakpoint is - I don't know how many stanza-fanouts  
> outweigh an RTT.
>
> I don't feel we know anything like enough about habits and metrics  
> right now. It could be that even when we have an RTT delay due to a  
> list rewrite on every event, it's not a big deal because the line  
> may still be efficiently utilised, but introducing new RTT delays  
> just doesn't sit right with me.

Do you  agree that in the case of "sending a message" though a  
repeater does not increase the RTT delay?

If yes, then this RTT delay is only present if I want to modify my  
remote list before sending a new message. Right?


> Now, I'm reasonably confident in suggesting that (without figures  
> to back this up), the list-change to stanza-fanout ratio has to be  
> pretty good before you beat simple compression.

Again, I think you are only thinking about bandwidth when you talk  
about compression. My concern is not bandwidth but how to deploy  
pubsub for hundreds of thousand subscribers without having my s2s  
links shutdown by big domains because I'm a spammer.

And the only way to do that is to allow the remote server to fan-out.

As for the list-change costs, and as suggested before by me with a  
pubsub use case, we might need to think about end-users being able to  
"add" themselves to a local repeater. This could be an option at  
repeater creation.

For example, if I'm a pubsub service at saramago.lit and I create a  
repeater at pessoa.lit for a specific node N, I can include a  
<allow_local_subscription /> to allow the list to be modified  
directly by users trusted by the repeater, or  
<allow_local_subscription><domain>pessoa.lit</domain></ 
allow_local_subscription> to restrict the ability of direct list- 
changes to a specific set of domains.

if this is allowed then, when people disco#info my node N, I can tell  
them that they can subscribe to me via a local repeater at their  
domain, and give the repeater jid for the list.

In this scenario, list changes are not from the pubsub service to the  
repeater, but by local users to his local repeater.

Security-wise, this is no worse than the previous version, because  
this capability is still controlled by the list creator, who has to  
opt-in to this.

RTT-wise if would be on par with a scenario without repeaters.

And we still get the benefits of remote fan-out.

Mind you that I see this possibility of end-users "adding" themselves  
to repeaters as a viable possibility for public pubsub nodes and  
auditorium-style MUC rooms only.


> I'm also of a much more nebulous gut feeling - assuming gut  
> feelings *can* be nebulous - that for typical large public pubsub  
> nodes - like, say, a popular blog - it's quite likely that the list  
> will often change between publication events - which in turn  
> suggests that this might be completely the wrong solution for that  
> use-case.

See my example of using a local pubsub service as a proxy, or the  
example above of letting repeaters list to be modified by user list.

I agree with you that list-change needs to be more streamlined, but I  
think we can do it inside the repeaters spec.


>> The cost of updating this list is exactly the same as a normal   
>> subscribe in pubsub. A second user will only see steps 1, 2, and  
>> 5,  same as a normal subscription. The local pubsub service might  
>> signal  the remote pubsub service, but thats optional, can be done  
>> out-of- sync with the user flow, and its only required if the  
>> central node  wants to have knowledge of the number of subscribers  
>> globally.
> Right, or control them, in a relatively trusted environment.

Even in a untrusted environment if you are talking about public  
information, like blog posts.

Trust, like, "I want to know if the notification is really from my  
friend Peter" is a problem for PKI and digital signatures, not the  
"multi-cast" system on top of XMPP. IMHO.


> A similar mechanism works for MUC, too - a sub-MUC would send  
> subscriptions to the master-MUC, which would send only a single  
> copy of the messages back to the sub-MUC for further routing. If a  
> user tries to subscribe to the sub-MUC and the subscription is  
> rejected by master-MUC, the rejection is passed back.

Exactly.

Again, I only see this as useful for auditorium-style MUC rooms. For  
normal MUC rooms, then you are implementing IRC on top of XMPP and  
multicasting message will be the least of your concerns (think  
netsplits, and making sure your nick is unique).


> We talked about this in Brussels a bit, and I think that it's an  
> interesting area to persue.

My opinion, is that without remote fan-out for pubsub, XMPP will not  
be a long term option as a notification network. Yes, it beats  
pooling, but source servers will be overwhelmed with the costs of  
sending the same notification to large numbers of subscribers.

Plus as a server-operator, its wasteful for me seeing thousands of  
the same message, with a different 'to' attribute.


> FWIW, pubsub and MUC - at least the public ones - are very much  
> simpler than presence handling, making this style of fan-out much  
> more practical. Indeed, having chains of sub-MUCs becomes  
> practical, since either the MUC is public, or potentially a chain  
> can be set up with bilateral agreements, which is effectively how  
> IRC and PSYC operate.

Yes, and I think that even if there is no consensus about repeaters  
as a general solution, we should have something to repeat public  
information via pubsub and certain types of MUC rooms.


>> For presence distribution, I like remote fan-out based on a  
>> reverse- roster better, but it doesn't interact properly with  
>> privacy lists.  Stanza repeaters deal better with privacy lists,  
>> because you can  update the remote list based on the privacy list  
>> changes. But I don't  know if the complexity of all this privacy- 
>> list interaction will be  worth it.
>
> Right - I think it's just going to get exceedingly complex, very  
> quickly, and the net result will be more bugs, and a failure to  
> produce scalable specialized support for fan-out in PubSub/MUC.

yep. I must admit that personally I would gladly sacrifice a remote  
fan-out solution for presence distribution if I could have one sooner  
for pubsub.

I say this because (if we exclude Peter) rosters above 1000 contacts  
are not that common (I'll try and see the distribution at SAPO) and  
the ones we found where mostly caused by bugs in the interaction with  
pyMSNt ;), but pubsub nodes with over 1000 contacts will become quite  
common if the social workgroup starts doing their thing.


> I'd like to know more about the levels of data we'd save for  
> presence fanout broadcasts, really, prior to going forward with  
> this, but I have to admit I'd be surprised if this is worthwhile  
> for just that.
>
>> In the end, a generic stanza repeater implementation is something   
>> useful to have in theory, but in practice all of the services that  
>> it  is trying to replace already have the subscriber list inside,  
>> and  they could leverage that to add multi-cast capabilities.
>
> Exactly.


Best regards,
-- 
Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: melo at simplicidade.org
Use XMPP!