[Standards] Council on Stanza Repeaters without Multicast
melo at simplicidade.org
Thu Apr 3 11:58:56 UTC 2008
On Apr 3, 2008, at 9:48 AM, Dave Cridland wrote:
> On Thu Apr 3 08:23:36 2008, Pedro Melo wrote:
>>> A) Compression
>>> It's crucial on two counts:
>>> 1) It's much simpler to implement, and2) Given that we are (or
>>> should be) encrypting every S2S connection, then TLS is giving
>>> us compression anyway, and moreover, it's cheaper to compress
>>> than not to compress.
>>> We've been down the road before where on some admittedly fairly
>>> simple figures I did a while back - and repeated for Peter's
>>> scalability draft in the SIMPLE WG, I think - the existing
>>> presence "splurges" compress really well, often better than the
>>> "optimized" alternatives.
>>> Smart Presence was one proposal that beat pure compression, as I
>>> recall, but it has security implications which I'm not going to
>> Bandwidth is not the only concern. If I have a very popular
>> pubsub topic (like a Atom node for some high-profile blog or
>> social site) with hundreds of thousands of subscribers, the cost
>> of expanding in the source is very high, in terms of CPU also.
> Well, yes and no - XML parsing appears to hit a lot of servers
> hard, but routing is routing, and repeaters will increase the cost
> there, I suspect. If you're having to expand routers, that
> expansion has to happen somewhere - it's a database lookup, or a
> filesystem read, or a hash table in memory - but it's certainly
> additional activity.
> This cost might, in some implementations, be outweighed by the
> saving in XML parsing. I don't know, and I don't pretend to know,
> either. I'm pretty sure it isn't in our implementation.
If you ignore list changes for a minute, routing costs will be the
same on remote nodes (the same amount of stanzas will be routed + 1,
actually) with or without repeaters. The biggest savings are in the
source service + source router which will have much less stanzas
transmitted and routed.
So globally, if you look at the cost of delivering a single
notification to thousands of subscribers, the cost to the XMPP
network as whole is lower with repeaters, and given the the fan-out
processing is pushed to the edges (and therefore the costs of lookup
are distributed amongst interested parties), I would be willing to
bet that the latency from source to destination will also be lower, a
So I do not agree that repeaters will increase the routing cost here.
Maybe a little on the remote hosts, but will save a lot on the source
As for XML processing, well, if/when that becomes a problem, we can
also use a xep-0138 profile for some more efficient representation
format for XMPP stanzas. For example, two ejabberd servers could use
native erlang message payloads when talking to each other... All is
fair you pre-declare.
>> The mailing list people moved to sub-lists to deal with this.
>> Repeaters are just like sub-lists. Please note that I'm talking
>> of the *public* pubsub node use-case.
> Right, almost. Repeaters are like sub-lists controlled by the
> sender - that's a quite strinking difference.
No, repeaters are sub-lists controlled by the service/the list. If I,
the sender, publish a node to some pub-sub service, I don't know nor
I even care if the pubsub service is using repeaters.
The sender has no control on the repeaters.
>>> B) RTT delays
>>> The introduction of Stanza Repeaters gives us RTT delays whenever
>>> a list needs changing. That just strikes me as worrying - I
>>> don't think anyone has clear figures on how much delay would be
>>> introduced, but I'm sure you'll agree that additional latency
>>> does not a performance improvement make.
>>> It may be possible to alter the protocol to remove these.
>> This is important if and only if list changes are the most
>> frequent operation.
>> If sending messages through the repeaters is the most common
>> operation, then the lower RTT of distribution that Stanza
>> Repeaters give us, will over-shadow the delays with list changing.
> Right, kind of - for any given repeater, you need a list-change per
> event to make it happen every time. And I'm not sure where a
> suitable breakpoint is - I don't know how many stanza-fanouts
> outweigh an RTT.
> I don't feel we know anything like enough about habits and metrics
> right now. It could be that even when we have an RTT delay due to a
> list rewrite on every event, it's not a big deal because the line
> may still be efficiently utilised, but introducing new RTT delays
> just doesn't sit right with me.
Do you agree that in the case of "sending a message" though a
repeater does not increase the RTT delay?
If yes, then this RTT delay is only present if I want to modify my
remote list before sending a new message. Right?
> Now, I'm reasonably confident in suggesting that (without figures
> to back this up), the list-change to stanza-fanout ratio has to be
> pretty good before you beat simple compression.
Again, I think you are only thinking about bandwidth when you talk
about compression. My concern is not bandwidth but how to deploy
pubsub for hundreds of thousand subscribers without having my s2s
links shutdown by big domains because I'm a spammer.
And the only way to do that is to allow the remote server to fan-out.
As for the list-change costs, and as suggested before by me with a
pubsub use case, we might need to think about end-users being able to
"add" themselves to a local repeater. This could be an option at
For example, if I'm a pubsub service at saramago.lit and I create a
repeater at pessoa.lit for a specific node N, I can include a
<allow_local_subscription /> to allow the list to be modified
directly by users trusted by the repeater, or
allow_local_subscription> to restrict the ability of direct list-
changes to a specific set of domains.
if this is allowed then, when people disco#info my node N, I can tell
them that they can subscribe to me via a local repeater at their
domain, and give the repeater jid for the list.
In this scenario, list changes are not from the pubsub service to the
repeater, but by local users to his local repeater.
Security-wise, this is no worse than the previous version, because
this capability is still controlled by the list creator, who has to
opt-in to this.
RTT-wise if would be on par with a scenario without repeaters.
And we still get the benefits of remote fan-out.
Mind you that I see this possibility of end-users "adding" themselves
to repeaters as a viable possibility for public pubsub nodes and
auditorium-style MUC rooms only.
> I'm also of a much more nebulous gut feeling - assuming gut
> feelings *can* be nebulous - that for typical large public pubsub
> nodes - like, say, a popular blog - it's quite likely that the list
> will often change between publication events - which in turn
> suggests that this might be completely the wrong solution for that
See my example of using a local pubsub service as a proxy, or the
example above of letting repeaters list to be modified by user list.
I agree with you that list-change needs to be more streamlined, but I
think we can do it inside the repeaters spec.
>> The cost of updating this list is exactly the same as a normal
>> subscribe in pubsub. A second user will only see steps 1, 2, and
>> 5, same as a normal subscription. The local pubsub service might
>> signal the remote pubsub service, but thats optional, can be done
>> out-of- sync with the user flow, and its only required if the
>> central node wants to have knowledge of the number of subscribers
> Right, or control them, in a relatively trusted environment.
Even in a untrusted environment if you are talking about public
information, like blog posts.
Trust, like, "I want to know if the notification is really from my
friend Peter" is a problem for PKI and digital signatures, not the
"multi-cast" system on top of XMPP. IMHO.
> A similar mechanism works for MUC, too - a sub-MUC would send
> subscriptions to the master-MUC, which would send only a single
> copy of the messages back to the sub-MUC for further routing. If a
> user tries to subscribe to the sub-MUC and the subscription is
> rejected by master-MUC, the rejection is passed back.
Again, I only see this as useful for auditorium-style MUC rooms. For
normal MUC rooms, then you are implementing IRC on top of XMPP and
multicasting message will be the least of your concerns (think
netsplits, and making sure your nick is unique).
> We talked about this in Brussels a bit, and I think that it's an
> interesting area to persue.
My opinion, is that without remote fan-out for pubsub, XMPP will not
be a long term option as a notification network. Yes, it beats
pooling, but source servers will be overwhelmed with the costs of
sending the same notification to large numbers of subscribers.
Plus as a server-operator, its wasteful for me seeing thousands of
the same message, with a different 'to' attribute.
> FWIW, pubsub and MUC - at least the public ones - are very much
> simpler than presence handling, making this style of fan-out much
> more practical. Indeed, having chains of sub-MUCs becomes
> practical, since either the MUC is public, or potentially a chain
> can be set up with bilateral agreements, which is effectively how
> IRC and PSYC operate.
Yes, and I think that even if there is no consensus about repeaters
as a general solution, we should have something to repeat public
information via pubsub and certain types of MUC rooms.
>> For presence distribution, I like remote fan-out based on a
>> reverse- roster better, but it doesn't interact properly with
>> privacy lists. Stanza repeaters deal better with privacy lists,
>> because you can update the remote list based on the privacy list
>> changes. But I don't know if the complexity of all this privacy-
>> list interaction will be worth it.
> Right - I think it's just going to get exceedingly complex, very
> quickly, and the net result will be more bugs, and a failure to
> produce scalable specialized support for fan-out in PubSub/MUC.
yep. I must admit that personally I would gladly sacrifice a remote
fan-out solution for presence distribution if I could have one sooner
I say this because (if we exclude Peter) rosters above 1000 contacts
are not that common (I'll try and see the distribution at SAPO) and
the ones we found where mostly caused by bugs in the interaction with
pyMSNt ;), but pubsub nodes with over 1000 contacts will become quite
common if the social workgroup starts doing their thing.
> I'd like to know more about the levels of data we'd save for
> presence fanout broadcasts, really, prior to going forward with
> this, but I have to admit I'd be surprised if this is worthwhile
> for just that.
>> In the end, a generic stanza repeater implementation is something
>> useful to have in theory, but in practice all of the services that
>> it is trying to replace already have the subscriber list inside,
>> and they could leverage that to add multi-cast capabilities.
XMPP ID: melo at simplicidade.org
More information about the Standards