[Standards] s2s and gracelessly broken streams

[JD Conley]

> > Assuming around a 60k limit because of memory limitations
> > on 32-bit architectures implies that each TCP connection
> > uses something approaching 30kBytes, which seems,
> > somehow, rather high to me.
> 
> It depends on a number of factors. For example, in Windows (x86), the
> limitations are imposed by the size of the Non-Pages Memory Pool deep
> in
> the Windows Kernel. Each async socket that's opened reserves a little
> bit of this memory, a when this memory is exhausted, you're out of
> luck.

OK, this is now WAY off topic. :)

We've actually done quite a bit of testing, both lab and real world, on
this stuff. A default install of 32 bit Windows box with >= 1GB RAM will
create roughly 150k sockets (it has 256MB non-paged kernel memory
available[1]). It's easy to test on your machine, just create a bunch of
sockets and watch the memory pool performance counter hit a wall. This
sounds pretty similar to the Linux figures, and of course varies based
on the buffer size you use for your socket, loaded drivers, and other
kernel level stuff that use the same pool.

The bottlenecks we ran into were more TCP stack and application level
limited, regardless of processor architecture. The amount of things your
average XMPP server does for a given connection puts the practical limit
around 100k until you start clustering, especially under failure modes
-- let's see how fast you can log-in 100k users after a network failure
with a single XMPP server...

OK, back on topic.

S2S connections should be able to be closed, like they are today. IMHO
C2S connections should as well. It's a good conservation of resources.
TCP stacks still suck at hundreds of thousands of simultaneous
connections, but they're getting better. I imagine with the multi-core
trend we're going to see much better stacks in upcoming kernels.
 
-JD

[1] http://support.microsoft.com/kb/294418