[Standards-JIG] Internet-scale subscription protocols

Ralph Meijer jabber.org at ralphm.ik.nu
Sun Nov 5 14:09:03 UTC 2006

On Sun, 2006-11-05 at 16:36 +1000, Benjamin Carlyle wrote:
> G'day,
> I work for Westinghouse Rail Systems Australia as a software developer
> and architect. I work in the SCADA world where soft real-time
> notification of the state of field equipment is central to the way our
> software operates. It is my goal that we should be using standard
> protocols for pub/sub. Ideally, protocols that are designed for the
> Internet scale.
> I have been trying to get my brain into JEP-0060 from the perspective of
> the more HTTP-oriented subscription patterns I have worked with and read
> up on in the past. In particular, Rohit Khare's dissertation on
> decentralising REST. The XMPP approach jumps the rails a bit from what I
> am used to and I would like to understand whether or not it meets my
> requirements.

Before going into your questions below, how, in your view, does XEP-0060
(we renamed all our extension protocols to XEP recently) jump the rails?

> My two fundamental requirements are:
> 1) That it must be scalable both in terms of how a communications
> end-point manages its direct subscriptions and how layering is used to
> divide and conquer the notification problem.

XEP-0060 as it is currently worded does not address this use case. It
assumes a world with three kinds of entities, publishers, subscribers
and publish-subscribe services. The publish subscribe service has these

 * Keeping topics (nodes) with their associated subscriptions
 * Do access control on nodes (using affiliations), on who can publish
   to which nodes, who can configure and administer nodes, and who is
   outcast from subscriptions.
 * Contacting node owners for approving subscriptions (when this is
   required by the node's configuration)
 * Upon receiving publish requests, handle the dissemination of the
 * Allowing access to previously published items, including payload, for
   so-called persistent nodes.

What you describe is a system that has a structure (most probably a
tree) of entities that disseminate published items. The origin entity
publishes to its 'home' publish subscribe service, and there are
intermediate entities that receive notifications on one hand, and
republish the items to send notifications to its subscribers.

While XEP-0060 doesn't describe this use case, I don't think we need
more protocol to accomplish this, though. A number of people in our
community have been looking at systems with a distributed notification
pattern, but there is no specification that neatly outlines how this
could be implemented. Peter and I should come up with a XEP on this,
probably, as it is very interesting for many use cases, including

> 2) That is be capable of dealing with notifications that come in faster
> than they are being delivered without a meltdown.

This seems like an implementation issue, not a protocol issue.

> I have written up a draft describing how this might be done for HTTP:
> http://soundadvice.id.au/blog/draft-carlyle-sena-00.txt
> and have an introductary blog entry here:
> http://soundadvice.id.au/blog/2006/11/02#introducingSENA
> I am particularly interested in how JEP-0060 achieves summarisation, the
> organised discarding of undeliverable information during overload
> conditions. My full list of features I am looking for (from the draft)
> goes something like this:
>      Summarisation. This is the organised discarding of older
>      information to ensure that slow clients recieve as much fresh data
>      as their connection characteristics permit.

An implementation may do some kind of summarization for specific
subscribers. XEP-0060 allows for subscription configuration options
(even custom ones) that would for example limit the amount of
notifications sent out for a particular node.

But I suppose summarization is dependent on what type of information you
are publishing. For example, if you have a node that publishes news
items (maybe using Atom), I could imagine digests of multiple entries in
one notification. But this limits the ability to access retrieve
distinct entries as publish-subscribe items.

For another type of data, the current ambient temperature for example,
it could be sufficient to just send less notifications for particular

>      Differential flow control. A slow client should not prevent fast
>      ones from getting updates.

XEP-0060 notifications are sent using <message/> stanzas. These are
one-off messages with no acknowledgments. So slow clients should not
influence fast ones.

>      Localised resynchronisation. A client need not reach back to the
>      origin server for the current resource status if its immediate
>      server is already handling the subscription.

This would be one requirement for the to-be-written specification.

>      Patch updates. For large resources (especially lists), the ability
>      to deliver a message that indicates the change from last time,
>      only. Not the whole state.

XEP-0060 allows for notifications without payloads. If a payload was
provided with the publish, and the node is configured to be persistent,
an entity may retrieve the payload at a later time by sending a request
with an item identifier that is sent with the payloadless notification.

>      Dial-back. Pub/Sub can be an amplifier of denial of service
>      attacks. The subscription mechanism must be able to detect
>      when its notifications are being treated as spam and end the
>      subscription.

I suppose much like mailing list software, an implementation of a
publish-subscribe service could remove subscriptions for entities that
have repeatedly sent back errors in response to a notification. Is that
what you mean?

> Is JEP-0060 what I want? Can it be pushed in a direction consistent
> with what I want? If not, is there some way to achieve constructive
> engagement between an xmpp-based approach and a non-xmpp-based
> approach?

Probably and Yes and Not-applicable but most likely yes.

As a last remark I want to point out that XEP-0060 should probably play
hand-in-hand with more general specifications such as Advanced Message
Processing (AMP, XEP-0079) for expiring notifications or unreachability
handling, Extended Stanza Addressing for multicasting identical
notifications to entities that share a 'home' server, and the Jingle
suite for doing out of band transfers of large payloads (maybe similar
to Publishing SI Requests (XEP-0137)).



More information about the Standards mailing list