[Standards-JIG] JEP-0017 Framing: Why was it rejected?
bob at wyman.us
Fri Oct 22 12:49:26 UTC 2004
Ralph Meijer wrote:
> Reading your comments was interesting.
Hopefully, you mean "interesting" in a positive way rather than as
simply providing comic relief... :-)
> I am guessing you are looking into framing for pubscom.com, and the
> outlook on many, many connections and a high volume of data transfer.
Yes, of course. But, the problem should be characterized as:
1. Many connections
2. High volume of transfers (i.e. stanzas per second)
3. Large stanzas
The result is that in an application of the type we build, there are
"many" simultaneous, multi-packet stanzas being processed at any one moment.
This is a classic condition for introducing framing in order to facilitate a
variety of efficiency (CPU + memory utilization) enhancements in the server.
> Have you, due the current lack of framing, experienced any issues so far?
Yes. Our current system, while completely capable of handling
current and near-term anticipated loads, does not achieve the throughput
that I expect. Of course, I am very well aware that my expectations may be a
bit drastic and old-fashioned... I'm an "old-fart" that grew up in an
environment in which hardware resources were very expensive and which
demanded that we were able to support hundreds of sessions on one-MIP
machines. (20 years ago on VAX hardware...) Given that today we're running
multi-Giga-Hertz machines with massive amounts of memory, I'm a bit stunned
by the inefficiencies of the system. Also, independent of the actual
throughput numbers, I can see all sorts of ugliness in the logic that seems
to be required to process unframed packets. Many, many techniques for
increasing or smoothing throughput simply don't seem to be useable with such
a protocol. This is disturbing. On the other hand, while I've built dozens
of distributed applications in the past, this is the first time in my entire
career that I've ever had to deal with such large volumes of unframed data
in this kind of an application. We can certainly do a better job then we
currently are and we're learning how to do it. What I'm trying to understand
is: "Will 'better' be good enough?"
I'm also interested in the fate of JEP-0017 since it proposed a
counted frame method which was very much like that which was adopted for
HTTP 1.1. (See: "Transfer-Encoding: chunked" in RFC2068 at Section 3.6,
etc.). The similarity between the rejected JEP-0017 framing and HTTP 1.1
framing is interesting since JEP-0124, while it claims to prefer HTTP 1.1,
does not take the "obvious" step of exploiting chunked encoding. The result
is that JEP-0124 implementations will be constantly burdened by TCP/IP
slow-start, a need to be create many more TCP/IP sessions then would
otherwise be necessary, and the requirement to waste network bandwidth due
to the need to repeatedly send HTTP headers that don't add any real
information to the session. In part, my questions about JEP-0017 are an
attempt to gain some insight into the thinking that might have influenced
the choices made in writing JEP-0124. But, a detailed discussion of JEP-0124
probably belongs in another message...
In summary, "Yes", I see problems in the running code that look like
they would be reduced if framing was in place. While I'm not concerned about
handling the loads we anticipate, it looks like it will take more hardware
then I would like. However, I'm not making any proposals right now or
advocating a position. While I'm generally prejudiced towards supporting
framing, right now I'm just trying to understand the issues, attitudes, etc
that have gotten us to where we are.
More information about the Standards