[Standards-JIG] JEP-0138 (Stream Compression): A case for LZO / even better algorithms

Peter Saint-Andre stpeter at jabber.org
Wed Nov 2 20:16:53 UTC 2005


Hi Andreas,

Thanks for the information. I don't think JEP-0138 will recommend one 
algorithm over another except for making zlib mandatory to implement 
(mostly because it is in common use). Some quick research on LZO does 
indicate that it is very fast, however, it seems to be a library that 
uses multiple algorithms rather than a well-documented algorithm itself. 
For instance, see the LZO documentation and FAQ:

http://www.oberhumer.com/opensource/lzo/lzodoc.php
http://www.oberhumer.com/opensource/lzo/lzofaq.php

I think we'd prefer to reference a somewhat standardized algorithm (as 
we do for zlib and lzw) rather than just a library.

Am I missing something here?

Peter

Andreas van Cranenburgh wrote:
> I don't know if adding yet another compression algorithm is a problem,
> but my limited knowledge says let's add LZO. This (possible lengthy)
> message argues why I think yet another one might be desirable / better.
> 
> 
> 
> When I originally mentioned my [rather naive] optimised Jabber
> compression idea (which was LZW-like as was pointed out to me), I hadn't
> really investigated the theory of compression. But now I read a little
> more, and came up with this:
> 
> Some details of the algorithm: http://en.wikipedia.org/wiki/LZO
> A speed comparison: http://compression.ca/act/act-summary.html
> 
> LZOP is pretty much the all-round fastest de/compressor, AFAIK. (note
> that this last link was last updated 2002...). 
> 
> Limiting bandwidth usage seems pertinent to (possibly busy) servers that
> want to provide any sustainable/scalable form of compression to clients.
> please think eg. of clients/servers in South Africa, that can only
> use ISDN lines (probably relatively expensive). IIRC dialup/ISDN is
> generally the fastest landline connection possible on the African
> continent.
> 
> 
> Replacing XML with anything else (such as ASN.1) doesn't really sound
> like an alternative to stream compression, to me, since there's also so
> much to gain by compressing all those message bodies a little/lot!
> 
> Even if one designes a very efficient replacement for the streaming XML
> part, you would end up with something where most of the data is spent on
> message bodies, and why not compress everything in the first place then?
> [in terms of compression ratio and implementation complexity, since
> speedups could theoretically be gained by avoiding the bulk of the XML
> parsing, at the cost of complexity/flexibility]
> 
> 
> 
> But before I second guess myself again, let the criticism fly around
> please. Is there any compression guru who knows what algorithm to use
> for (streaming) XML? A benchmark also seems in place, if anyone knows
> how to do a proper one (ie. compress real-life data, in real-time,
> monitor CPU usage + comp. ratio for each algorithm concurrently).
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3641 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://mail.jabber.org/pipermail/standards/attachments/20051102/33e3e1b9/attachment.bin>


More information about the Standards mailing list