[jdev] resources available via XMPP... [long]

Jeff McAdams jeffm at iglou.com
Tue Apr 22 09:00:26 CDT 2008

Ernest Nova wrote:
> Let's see if I can summarize for my purposes :) Longish response
> follows excerpt.

Yeah, as we get deeper and deeper, the messages keep getting longer.
Makes for a good discussion of the ideas, though.

> OK - I am in violent agreement with your ultimate choice, but not
> entirely satisfied by your reasons :-)

Oh yes, I think we're agree in general, only differing on details.

> Given the nature of your application category (a variant of
> public-safety/ first-response), I suppose my reaction is one of:

Well, do be aware that, while the main IEMChat is focused on
interactions between the NWS, and the media and Emergency Management; my
approach to this discussion has been more about the publicly accessible
side of IEMChat (https://iemchat.com/public.phtml )

Because this public side is less about collaboration and first-response,
I'm focused, for the purposes of this discussion, on the real-time
nature of the notifications, as that's what I think will draw the public
in to using an XMPP based system rather than HTTP polling an RSS feed.
Yes, only some events really *need* that real-time response, but since
its also better for the information providers (no HTTP polling), overall
I think its a great way forward for everyone.  Its just a matter of
convincing people to adopt a new'ish technology to do it, and figuring
out how best to use that technology.

And I'll come back to it being intended for public consumption later as
well.  :)

> "But, but .. you are not even using *all* the good parts".  

Oh, no doubt on that.  This stuff is all still in its infancy and huge
benefits are there for the taking as soon as someone that has the
programming chops to do it gets the round tuits available to do the work
to make it happen.  Thus my involvement here, to try to help make people
aware of the cool possibilities that exist in this space and to try to
draw some development effort to it to do cool stuff.

> You say, in your experience, speed of notification has been more
> important than subsequent collaboration in first-response. I was
> arguing that they may be equally important and use of XMPP or any
> presence-enabled IM service SHINES most when an application can weave
> in both aspects: low-latency delivery and built-in collaboration;
> first-response being one of those situations.

Well, let me qualify some.  For the main IEMChat...the collaboration
aspect of it is huge, indeed, that's the main reason for its existance.
 iembot, with its real-time feed, is the treat that helps draw
participants into the project.  Virtually all of the participants have
some sort of near-real-time notification of these alerts besides iembot
in any case.  iembot and its real-time notifications are certainly
convenient, but not critical on the "real" IEMChat side.

On the public side, however, that's flipped around.  Many people in the
public do *not* have that sort of a near-real-time feed of these events.
 The best notification system for the general public (in the US, at
least) is NOAA Weather Radio, and the existence of iembot doesn't change
that.  I'm trying to make iembot and the public IEMChat available as an
additional notification capability, and a notification capability that
is more easily integrated into other systems.

So, I think the public notification is more about timeliness than it is
about collaboration, while the "real" IEMChat is more about
collaboration than it is the notification timeliness.  In both sides,
however, both aspects are great features of the system.

> Like all things in life that comment about first-response is an "it
> depends" item. "Information" in the face of an imminent or ongoing
> public safety event is both "system-generated" and "user-generated".
> For example, having had a house in the line of the California fires
> last year, a time-sequenced flow of information from applications
> immediately validated by first-person accounts, such as is presented
> naturally ordered in a chat room, possibly moderated and geo-location
> tagged, would have been of most help rather than asynchronous
> notifications. 

Whole-heartedly agreed.  I'd love to see things get geo-location tagged
as well.  :)

> Likewise, a "run for your lives" event requires prompt notification and
> initially probably not much else but I would argue that you are not
> always going to be sitting around  in front of a computer ( for
> computer, read richly-connected device). The need would be to deliver
> the alert [and subsequent updates] to personal mobile devices in
> parallel and an IM application may not always be the fastest
> notification delivery mechanism on mobiles. It is still the the best
> app when you need people to  people communication following
> notification.

Again agreed.  Anytime that I talk about IEMChat, I try to always stress
that people should still have Weather Radios in their home or business.
 Notifications to personal mobile devices is a great thing as well, but
for weather, NWR is still king for the best combination of reliability,
ubiquitous access, and timeliness.

> I should note that when building for pure notification use cases, we
> ended up building alert delivery via direct one-to-one IM , that is
> direct messages from the bot to each end user, rather than via chat
> rooms. This was mostly because, as you have also alluded to,  chat room
> implementations have not yet been designed to scale for broadcast
> functions. In large populations, they can also keel over just with the
> normal join/leave traffic even when no messages are being posted to the
> rooms.  Also, not every end-point will be on XMPP chat rooms or even
> pub-sub and other delivery mechanisms (SMS, voice notifications) are
> easier to support with one-to-one messaging.

Once more agreed, and there's another issue.  Coming back to the idea of
a significant earthquake in the mid-west.  The last time that an
earthquake in this region of magnitude greater than 5 occurred was 1968.
 So, we're dealing with events that are infrequent enough that sitting
around in a chatroom to be alerted to an event that may not happen for
40 years is not exactly feasible.  So I'm thinking about the ability to
register for a notification that will be pushed to your regular way of
being notified of events in general...that may be a private
bot-to-person IM message, or maybe something like pushing it into Mac's
growl notification system.  Many possibilities exist here, and this is
another example where a chatroom isn't the way to go unless its
piggy-backed on a chatroom that does have more frequent updates
happening in it, such as weather events...and then we're back to the
overloading of notifications and other stuff that we've talked about

> IM servers do act as fast message switches - so we found replicating
> messages "outside" the IM server and then pumping them through the IM
> system resulted in the fastest delivery (as compared to using the
> replication capability of the chat room.  We also side-stepped the
> group presence-traffic this way.  

Interesting...do you have any published documentation on this?  I'd be
interested in reading up on it and considering the implications of this.

> It might offend some, but pub-sub is at high functional level 
> essentially an access-controlled broadcast chatroom without the
> presence traffic/ room roster.

Oh, absolutely.  I frequently wonder, if had pubsub been developed
first, would GroupChat and MUC have been developed as a special case use
of pubsub rather than the much more separate facility that it is.  I
also wonder, if that had happened that way, would we be seeing much
greater adoption of pubsub in general than we are today.  *shrug*  What
might have been?  Ah well, we've got what we've got.

> The message replication performance and
> scalability there too will depend on the implementation. You do get the
> benefit of light weight topic/subscription management - but in any
> sophisticated alerting system with complex subscription rules, these
> rules would need to be populated outside the IM-system using a more
> visual interface. You may also lose the benefit of presence-based
> dynamic routing of alerts.

Yup, raw pubsub is nigh on unuseable for a basic IM client.  I'm not
sure what needs to happen on the client side, but clearly clients need
some sort of way to receive pubsub notifications and do something
intelligent with them if XMPP pubsub is to ever really take off as a

> I agree that large-scale notification systems on the internet, as you
> say,  "suck" - but as an IM-biased-developer the next part is going to
> pain me even more to say.. If all you *really* want is super-fast
> internet scale notification, and the collaborative part is, as you say,
> a "shrug"; Then, mister, I would argue then that XMPP, even with
> pub/sub, is in fact heavyweight for notifications. XMPP is not a
> lightweight protocol (comparisons to SIP not withstanding) and one
> could build better optimized and scalable systems for first-response
> one-way notifications.  

Yes, you are correct here, but this is where I come back around to my
emphasis for the public consumption.

The fastest, real-timeiest, notification system would almost certainly
be a very low level, domain specific protocol on top of a simple tcp (or
udp) socket in a dedicated client.  And, indeed, this is exactly what
the NWS and USGS do for their systems (NWS uses Unisys' LDM, USGS has
their QDDS).  For public accessibility, however, (and I think we'll
agree on this) a dedicated client for each type of notification just
isn't feasible, and building on top of XMPP has a more ubiquitous
transport is such a great win as to be pretty much a no-brainer.  All of
the issues and aspects that we've talked about make XMPP a fantastic
transport for these sorts of capabilities.  The collaborative abilities,
the real-timedness (I just love making up variants of that term), the
relative ubiquity compared to dedicated clients, and the omnipresence to
be able to slip-stream rare events in to a notification channel that
exists for many things, not just the rare events; all of these aspects
combine to make XMPP a great transport for the combination of needs.

> You may for example be better served by a stripped down tcp-socket or
> other ordered-delivery server that does little else but initial
> authentication and leaves the pipe open for pushing further data down
> to the client. The client could then be integrated with our non-IM apps
> such as an RSS aggregator or an embedded browser component to collect
> related information. Even certain readily available JMS systems, by
> definition pub/sub servers,  have much higher message switching
> throughput and lower latency, and certain web-streaming technologies
> can accomplish higher speeds and number of simultaneous connections
> over web browsers (search for the financial tick data delivery)

Performance is not, yet, a problem.  It may grow to the point where it
will be an issue...but if it does, I believe that there will be enough
visibility and interest in the issue that solutions will be found.  :)

> That said, no matter how you came to pick XMPP for public safety
> notifications given faster alternatives, over time you will appreciate
> your choice even more for *all* that it offers. I believe most urgent
> notification will need to be followed by a multi-channel collaboration
> or a closed-loop response.

Yup, we certainly are in agreement.  As I said, IEMChat is still young,
and the capabilities are still maturing.  There's so much more that can
be done with it, and hopefully this thread has helped raise the
awareness of people about that project and the possibilities presented
for other sorts of things along those lines (including earthquake alerts
sourced from the USGS QDDS system ;)

I should also say that I am not the mastermind behind IEMChat.  I didn't
start it, I don't do the bulk of the day to day running of it.  I *am*
on the admin team of it, so I do help in maintaining it and running it,
but I wasn't the originator.  I was brought on board and asked to be a
part of the admin team because of my experience and knowledge of XMPP
based systems (which IEMChat, obviously, uses).  I'm something of an odd
bird on the IEMChat system because I'm nearly unique in my almost
complete lack of meteorological background and knowledge.

Anyway, its great to bounce the ideas around.  To come back to square
one, anyone interested in playing with QDDS?  :)

Jeff McAdams
"They that can give up essential liberty to obtain a
little temporary safety deserve neither liberty nor safety."
                                       -- Benjamin Franklin

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 249 bytes
Desc: OpenPGP digital signature
URL: <http://mail.jabber.org/pipermail/jdev/attachments/20080422/52972362/attachment-0002.pgp>

More information about the JDev mailing list