[jadmin] Scalability - Commercial solution the only way (?)

[Peter Saint-Andre]

Paolo Perazzo wrote:
> my understanding is that 
> if you wanna have a very scalable solution, MSN, Yahoo, AIM like to 
> support millions of users, the only solution today is going through a 
> commercial solution. Ejabberd seems to be the closest open source 
> solution, but still not ready for that probably (again, correct me if 
> i'm wrong)

Very few organizations need to support millions of users. And those who 
do tend to have the money to either build their own solution (cf. 
Google) or purchase a commercial solution (e.g., really big ISPs). Also, 
do you mean millions of registered users or millions of concurrent 
users? In my experience, usually only 2% or perhaps 5% of registered 
users are online at any one time, which means that even fewer 
organizations need to support millions of concurrent users. I suppose 
only AOL, MSN, Yahoo, and Google -- do you work for one of those? ;-)

> - Do you have any suggestion for commercial solutions that can scale 
> that much?

The only commercial solution I know of that will perhaps scale that high 
is Jabber XCP (from Jabber Inc.), which has been deployed at large 
service providers like Orange, Wanadoo, and BellSouth.

> - Do you know btw what google talk is based on? They talk about jabber 
> server, but it might be a generic term

Google wrote their own solution.

> - Beside "Jabber Server Farming How-To", 
> http://www.faqs.org/docs/Linux-HOWTO/Jabber-Server-Farming-HOWTO.html, 
> is there any effort in the open source community to design a scalable 
> solution, in terms of software and hardware architecture. Is the effort 
> big or it's just a matter of replicating the servers on multiple 
> machine, have some load balancing and virtual server functionality in 
> front of it (I know i'm simplifying) or it's something more complicated? 
> What are the pieces that are missing today? I think this would be a very 
> interesting task, from an architectural and design perspective.

One of the core principles of open-source development is that most 
projects are started by someone who wants to "scratch an itch" or solve 
a problem that is personally important. While it may be architecturally 
"interesting" to design and implement a massively scalable IM server 
solution, this has never really been an itch that any Jabber developer 
or team has urgently needed to scratch, or even (with all due respect to 
Jer, Rob Norris, Alexey Shchepin, etc.) been able to scratch -- partly 
because it's hard to design a massively scalable XML router (hint: it 
would probably help to have quite a bit of Unix kernel development 
experience), partly because to really test such a solution you'd need 
access to the kind of big hardware that your average open-source 
developer does not have access to, and partly because designing 
massively scalable servers is something that is usually a corporate 
objective and not a personal objective.

I'm not discouraging you from starting such a project or joining an 
existing effort that has a chance to create a massively scalable 
open-source solution (e.g., ejabberd or Jive's Pampero). But IMHO you'd 
to better to get involved with one of those projects than for us to talk 
about it on this list. :-)

> - Do we know how the big guys are implementing their scalable solutions?

By "big guys" do you mean AOL, MSN, and Yahoo? I don't know much about 
their implementations / deployments, but I understand they have lots of 
hardware...

Peter

-- 
Peter Saint-Andre
Jabber Software Foundation
http://www.jabber.org/people/stpeter.shtml
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3511 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://mail.jabber.org/pipermail/jadmin/attachments/20051011/305573c5/attachment-0001.bin>