[standards-jig] JNG Ramblings.

Mike Lin mikelin at MIT.EDU
Mon Aug 12 07:06:32 UTC 2002

Hi Nathaniel, thanks very much for your most lucid comments.

I think you're totally right on all counts. I would just like to expound
a bit on the MIME comment I wrote.

The objection I wrote about MIME is not that I thought it required
base64 payload encoding; it's that I don't feel it's appropriate as a
framing protocol for streaming data because of the loose textual nature
of the frame itself (textual headers, boundary strings, etc.)

The key goal that I'm trying to achieve with the wire protocol is that
it should be very easy to frame a message under the conditions of
asynchronous socket input: data may be received in arbitrary chunks and
there is no preordained knowledge of how much data there is to receive.
In contrast, a MIME message, to my understanding, is typically parsed
only once it is fully received.

The most flexible model for framer implementation under the asynchronous
conditions is as a reentrant state machine that accepts (or reads
directly) data in arbitrary chunks and returns properly framed messages
once there is enough data available, but always returns control to the
caller whether there is enough data available or not. This is similar in
principle to a "pull parser" which, instead of blocking if there isn't
enough data available, returns control to the caller, saying to get more
data and try again.

When you have to read text headers and boundary strings without
predetermined length, it is very difficult to implement such a state
machine. For example, say that there is a MIME header "Length:
123456\n", but network congestion happens to work out so that you only
receive "Length: 123", with the "456\n" delayed to some later time. The
framer has to enter a state where it has received part of the length
header, and is awaiting the rest, then return to the caller, prepared to
continue lexing the integer when more data comes in. Permuted over all
the other possibilities for header and boundary string fragmentation,
this leads to a very large number of states that must be explicitly

In contrast, the protocol I'm experimenting with has fixed length
headers. The framer knows it has a header as soon as it has 4 bytes; so
the number of states in waiting for a header is very small. You either
have 4 bytes and are ready to proceed, or you don't. Those 4 bytes tell
you how many more bytes to read to get to the next 4-byte header. This
simplicity in the number of states makes it very practical to implement
in any programming language; I have a couple hundred lines of Java code
that I wrote in the space of an hour or two that does it, and the
non-event-driven programming model it enables is sweet, sweet nectar.

With such a simple protocol, it is even practical to model it with
automata theory, leading to framer algorithms that are provably correct,
where "provably correct" is used in the strongest mathematical sense of
the phrase. When we have such a high level of confidence in correctness,
it is reasonable to talk about putting the framer down low in the
kernel, in order to achive ludicrous performance. Anyway, I will not
speak further of this arcanity. :-)

In conclusion, the key reason that I feel MIME is inappropriate at this
level is that it is too hard to write something to parse MIME under
asynchronous input conditions. The "MIME is really heinous" comment was
a bit below the belt and I think it is just a reflection or our focus on
XML, which unfortunately has become a bit obscured in all this

Footnote: The above also explains why I don't like the fact that BEEP
has variable-length text headers. I feel it unecessarily complicates
things for framer implementations, for questionable benefit. We have
tools like ethereal to see into the binary wire just like we have tools
like microscopes to see into, well, the microscopic. If I were in the
position of having to write a BEEP implementation, I would be much
happier if BEEP had fixed-length binary headers, or at least text
headers with numbers left-padded with zeros so that they are fixed

Footnote 2: Current Jabber server implementors happen to be very
fortunate because Expat is designed in the state machine manner I
described above; data is handed to it in arbitrary chunks, so its
lexer/parser is fully reentrant in that it can be paused at any point
until more data is available. I consider this an exceptionally
impressive feat of software engineering, because in a rather low-level
language like C, it is extremely difficult to do. I recently wrote one
in OCaml, which unlike C has functional closures that make it
considerably easier. Besides these two, I do not know of any other XML
parsers that have this capability, although I imagine there are some for
Scheme and SML, whose first-class continuations vastly simplify this
task. Anyway, the result is that for most languages it is impossible to
mix asynchronous I/O with the current Jabber protocol without writing
your own fully reentrant XML parser.

Footnote 2a: The .NET parser seems to go halfway; it seems to be
reentrant up to tag boundaries, which is why Joe has to search for '>'s
as the chunking boundary in Jabber.NET. I don't blame Microsoft for
this; as I said, full reentrancy is very difficult to do in a
nonfunctional language, as C# is.

Anyway, I hope this clarifies some of the motivation for how my
/experimental/ protocol is designed.

Thanks for reading,

On Sun, 2002-08-11 at 15:47, Nathaniel Borenstein wrote:
> As a Jabber newbie, I've just been lurking on this list, but I can't
> resist joining the discussion about Mike's ideas, and especially the
> issue of a binary protocol on the wire.
> > When you combine the wire protocol and the manifest, you get something 
> > that does basically the same thing as MIME.  Yes, that's true. But 
> > it would be very messy to use MIME for wire protocol framing 
> > because of its loose textual nature. 
> This repeats a common misconception about MIME, which is that it is
> *inherently* textual.  That is certainly the way it is generally used,
> but it is not *necessary* to use it that way.  In particular, MIME was
> designed to support binary transport optionally, even for multipart
> messages.  The key is the separation of "Content-Type" from
> "Content-Transfer-Encoding" (C-T-E).  If you have a binary-capable
> transport -- which email definitely was NOT when MIME was designed, and
> still isn't, really -- then you can simply throw away the C-T-E (and
> base64) and still use MIME just fine.  
> The only tricky bit is that when using a multipart, you have to be
> careful that your boundary doesn't appear in the body of any of the
> objects being encapsulated.  (This is easier for base64-encoded data,
> because you can include in your boundary a character that isn't in the
> base64 alphabet.)  But you could even eliminate that problem by
> modifying MIME to add a "Content-Size" header.  That was much-discussed
> during the MIME design, but it ultimately was rejected as being
> incoherent for non-binary data over SMTP.  It should be fine for true
> binary data in a binary protocol, as long as nobody is quietly
> converting the data (say, between differing newline conventions) along
> the way.  
> > Besides, I think MIME is really heinous.
> If you think that MIME requires C-T-E's and 7-bit transport, I can
> certainly understand why you'd view it that way.  But if you eliminate
> that problem, I think it is rather less heinous in this case than having
> a maximum of 128 choices for the "Type" field and therefore an inherent
> need for a content-type mechanism underneath it.
> On the larger issue of whether a protocol such as Jabber's should be
> binary or textual, you might be surprised to hear the author of MIME
> coming down on the side of a binary protocol, but that's exactly what
> I'm doing, primarily for the reasons given by Richard Dobson in his
> message.  We did not design MIME as a textual protocol for ease of
> developing/debugging.  We designed it that way because of the
> constraints of real-world SMTP implementations that simply couldn't
> handle 8-bit or binary data.  I don't *think* Jabber should have to deal
> with similar constraints, although this may be an area where my
> newbie-hood is showing.  And while I sympathize with the folks who want
> to telnet directly, I don't think it would be very hard to write a
> telnet gateway that makes it easy to type (almost) directly to the
> server from a textual interface, and that's a small price to pay for a
> 33% speedup on big binary objects.  (Alternately, we could use an
> approach like ESMTP which makes binary transport optional, and this
> could be nearly always used by software but rarely or never by people
> connecting with telnet.)
> Unless there is a big need for backward compatibility with
> non-binary-capable software, I would strongly support at least a binary
> *option* in the protocol.  -- Nathaniel
> ----
> http://guppylake.com/~nsb
> ----
> _______________________________________________
> Standards-JIG mailing list
> Standards-JIG at jabber.org
> http://mailman.jabber.org/listinfo/standards-jig

More information about the Standards mailing list