[jadmin] Scalability - Commercial solution the only way (?)

ahecox at uchicago.edu ahecox at uchicago.edu
Wed Oct 12 18:42:22 CDT 2005

On Tue, 11 Oct 2005, Paolo Perazzo wrote:

>> From: ahecox at uchicago.edu
>> Reply-To: jadmin at jabber.org
>> To: Jabber server administration list <jadmin at jabber.org>
>> Subject: Re: [jadmin] Scalability - Commercial solution the only way (?)
>> Date: Mon, 10 Oct 2005 19:21:12 -0500 (CDT)
>> On Mon, 10 Oct 2005, Paolo Perazzo wrote:
>> > Thanks all for the interesting debate on the server options. I thought it 
>> > would have been good to create a new thread for this topic even if kinda 
>> > related to the previous. Reading around, my understanding is that if you 
>> > wanna have a very scalable solution, MSN, Yahoo, AIM like to support 
>> > millions of users, the only solution today is going through a commercial 
>> > solution. Ejabberd seems to be the closest open source solution, but 
>> > still not ready for that probably (again, correct me if i'm wrong)
>> > 
>> > - Do you have any suggestion for commercial solutions that can scale that 
>> > much?
>> > 
>> > - Do you know btw what google talk is based on? They talk about jabber 
>> > server, but it might be a generic term
>> > 
>> > - Beside "Jabber Server Farming How-To", 
>> > http://www.faqs.org/docs/Linux-HOWTO/Jabber-Server-Farming-HOWTO.html, is 
>> > there any effort in the open source community to design a scalable 
>> > solution, in terms of software and hardware architecture. Is the effort 
>> > big or it's just a matter of replicating the servers on multiple machine, 
>> > have some load balancing and virtual server functionality in front of it 
>> > (I know i'm simplifying) or it's something more complicated? What are the 
>> > pieces that are missing today? I think this would be a very interesting 
>> > task, from an architectural and design perspective.
>> > 
>> > - Do we know how the big guys are implementing their scalable solutions?
>> > 
>> How far do you need to scale? We're planning a jive deployment and tested 
>> to ensure scalability to 25k concurrent users with their software. A couple 
>> of JVM options and a 64 bit machine with plenty of memory and we were able 
>> to reach that goal with a relatively modest (< 10k) box.
> I think this would be a very good starting point, and to be honest I don't 
> mind having multiple domains and use s2s communications if I wil have to 
> scale more. What are the JVM options you tweaked btw?

The first problem we ran into was a lack of heap space: we solved this by 
manually increasing the heap size: we approximated heap growth using:

heap_size_MB = number_of_users * .109 + 18 for Sun's JVM
heap_size_MB = number_of_users * .113 + 27 for BEA's JVM

Since we ultimately decided on BEA's JVM, that meant tweaking -Xmx: & Xms: 
to a large size. However we couldn't push that too far or we'd run out of 
space for native threads. Since a 32bit JVM only has access to 31bit 
address space, the java threads, native threads, managed heap, and thread 
stack must all fit in under 2Gb. This was solved by using a 64bit JVM and 
a 64bit OS.

The larger the heap, the longer GC takes, so we used a generational 
garbage collection (-Xgc:gencon for BEA) but are (in production) keeping 
the heap only as large as it needs to be for our user base. While 25k is 
our max goal, we're only setting things to handle about 5k concurrent 
users right now. So to meet that:

-Xss:128k -Xms:1024m -Xmx:1024m -Xns:256m -Xgc:gencon

Scale Xms, Xmx, and Xns up linearly, depending on the actual number of 
concurrent users you have. I'm sure these could be optimized more, but 
these meet our performance criteria. Also, depending on your OS, you might 
run into some additional limits. We're running on Red Hat, so, for 
example, we had to use ulimit to increase the number of open file handles.


> thanks for sharing this info, the numbers in the comparison chart 
> (http://www.jabber.org/admin/jsc/) were pretty limited for Jive
> "about 100 concurrent users maximum online in internal deployment, maybe more 
> in environments we don't know of"
> Best regards,
> Paolo
>> Obviously their architecture will scale even better in the future, as 
>> Matt's already pointed.
>> -Andrew
>> _______________________________________________
>> jadmin mailing list
>> jadmin at jabber.org
>> http://mail.jabber.org/mailman/listinfo/jadmin
>> FAQ: http://www.jabber.org/about/jadminfaq.shtml
>> _______________________________________________
> _______________________________________________
> jadmin mailing list
> jadmin at jabber.org
> http://mail.jabber.org/mailman/listinfo/jadmin
> FAQ: http://www.jabber.org/about/jadminfaq.shtml
> _______________________________________________

More information about the JAdmin mailing list