[Standards] file transfer (was: Re: [LONG] Jeez, Sorry)

Peter Saint-Andre stpeter at stpeter.im
Mon Aug 27 22:46:59 UTC 2007


Please stop top-posting. It makes the discussion hard to follow.

Jonathan Chayce Dickinson wrote:
> I'm sorry. Didn't mean any harm. 

I know you didn't. I appreciate your enthusiasm, I really do! But let's
temper it with some realism. :)

> I'm not too hot on bandwidth here, 

I'm not too hot on mental bandwidth here, either. :)

> so I
> didn't come across MTOM.

So you didn't research very thoroughly, did you?

If you click "I'm Feeling Lucky" at http://www.google.com/ when
searching for "Direct Internet Message Encapsulation", you come to the
following page:

http://xml.coverpages.org/dime.html

And right there at the top you find this text:

Note: Microsoft has listed DIME among "Superseded Specifications" in its
Messaging Specifications Index Page. See Direct Internet Message
Encapsulation (DIME) [June 17, 2002]: "This specification has been
superseded by the SOAP Message Transmission Optimization Mechanism
(MTOM) specification."

So here we find many things we need to know. In particular, the
technology was superseded and was never seriously pursued. So why are we
discussing it here?

However, your suggestion might lead to a productive discussion about
file transfer / file sharing, see below....

> I wasn't wasting your time, quote: "Didn't want to push this forward
> until I got some feedback". I don't like wasting people's time, hell, I
> love helping people out and solving problems: don't reply to this and
> you will never hear of it again; all it takes is "nope, not a good
> idea," not something as violent as "fanciful" (ouch!).

It's fanciful to seriously suggest using a technology that was
superseded and abandoned over 5 years ago. Let's try to use more
standardized and widely-available technologies if possible.

> Read the spec and you will see that it has a lot to do with XMPP: Namely
> (quoted from my draft):
> 
> 1. DIME will rectify the absence of true In-Band Bytestreams, as it
> defines methods to send multiple payloads in a atomical message. Each
> payload in the message can reference other payloads in the message.
> 
> XMPP needs, in my opinion, a way to negotiate real in-band bytestreams.

What are the problems with XEP-0047?

> Base64 seems like a cop-out to me, it really does.
> 
> 2. Furthermore compression methods used by servers (which tend to be
> fast instead of -small-effective) will, at best, return the data to near
> it's original length. If the data is presented to these algorithms in
> its original state they may be able to compress it even further.
> 
> Not everyone has a T3 connection at home, some of us live in South
> Africa, and well, bandwidth here is rather expensive: and so is a static
> IP (yes, we differentiate between them here, we /actually/ have
> non-static ones too). When you have to weigh up the value of 1MB, you
> will know.
> 
> 3. Ultimately this protocol will smooth out the issues plaguing Jabber
> regarding file sharing.
> 
> This was a hot topic on JDev not too long ago! Demand, supply. If the
> developers are demanding it, I ASSURE you, the users will be fast
> following (faster than any cheetah metaphores ;)
> 
> 4. Almost all the existing XEPs should be able to derive some further
> benefit from this extension (especially Jingle and SI).
> 
> That is a bit fanciful and ambitious, I admit. Add XHTML to that list,
> and well there you go, not even nearly all, but 3, which is a start.
> 
> (Just coming in on that regard, SOAP has nothing to do with XMPP, but
> neither did SASL (at the start), or XHTML... And there is SOAP over XMPP
> now)
> 
> And, well, *both* are XML and *both* suffer from the fact that you can't
> package binary data reliably and efficiently in an XML document. 

This is very true.

Jabber was not designed for binary transfer. This is a strength for the
core use cases, but a weakness for things like file transfer and file
sharing and media sessions, as we have painfully discovered time and again.

However, personally I'd say we don't want to use an abandoned technology
to solve the problem.

Another approach would be to define a BEEP transport:

http://www.beepcore.org/

Yes, BEEP is not widely implemented either, but I at least there are
RFCs (and I am aware of a new push for wider implementation).

Maybe it would help if I finished writing up some results of the file
transfer discussions at the recent XMPP DevCon?

> MTOM
> wouldn't exist if it wasn't a real problem. It would have ended with DIME.

But why was DIME abandoned? (Not that I like MTOM, mind you!)

> A lot of protocols do it, hell, even your cellphone does something
> similar (which is why you can call someone and surf over GPRS/3G/HSDPA
> at the same time).
> 
> In regard to Dave's email, about the negative compression etc. very
> true, didn't think of that. Nice, constructive criticism. Dave: the very
> reason I thought of it was because a P2P connection isn't always
> possible, as in my case (I don't need it, but the awareness is there). I
> was hoping to use the server to host the Peer-To-Host-To-Peer connection
> instead of some 3rd party website/ftp server.

That appears to be a sensible line of thinking. However, it seems to me
that the wide availability of HTTP servers (which could be bundled with
your favorite XMPP server) makes an HTTP approach even more appealing.

Part of my question is: do we really need to stream files? Think about
the difference between media streaming and media downloading. The use
cases in which true streaming is needed seem few. Even things like
podcasting are not truly casted in real time -- they are uploaded to an
HTTP server and then downloaded, to be listened to on demand.

Similarly, it seems to me that if I want to send you a file, I could
upload it to an HTTP (associated with my XMPP server), get a URL for
that file, then send you the URL. If I don't have an HTTP server at my
disposal, I could fall back to in-band bytestreams. But let's face it,
the HTTP community has file-upload and file-download down cold. Why not
leverage all that? We don't need to do *everything* via XMPP! Do we
really need streaming here, or does it just seem cool?

> I saw a gap, jumped, it seemed like a good idea, but it obviously
> wasn't. 

Again, nothing is obvious. :)

The main point is to solve the file transfer problem once and for all. I
happy to think that the idea of using DIME to solve the problem is
far-fatched because DIME is such an obscure technology, but your
proposal might lead us in the right direction. So thanks for that. :)

> Sorry about putting your name on it Peter: it was in the
> template, I thought it belonged there. 

Oh gosh no, I'm not that egotistical. :)

> I'm sorry I put this out there, I
> could have just as easily made the whole thing a undocumented protocol
> on the project I am working on, which actually, ironically, REALLY needs
> it: and probably will land up doing now anyway.

Before diving all the way into DIME, please do reconsider.

> On that note, it was nice contributing to the Jabber scene, but I should
> really take my leave. I just thought the OSS/OSI community was made up
> of people with broader minds, a.k.a Dave. Peter just read the two emails
> and look at the elegant way in which Dave substantiated his response,
> instead of "your ideas suck".

I never said your ideas suck. I said the notion of using DIME (an old,
experimental, barely-hatched, superseded, abandoned technology) was a
bit fanciful. Let's try to use something that we can really build on.

> There are some things schools can not teach you about English, I read
> this email about 5 times before I sent it, asking myself, "how may I
> hurt the other person?" 

Don't take it personally. If you're going to put your ideas out there,
you need to develop a bit of a thicker skin.

That said, I'm sorry if you felt attacked.

I want to solve the file transfer problem as much as you do -- this has
been a thorn in our side for 8 years. But we need to do so in a
sustainable way, with common technologies, using protocols that have
pluggable libraries that client and server developers can make use of,
etc. Or so it seems to me.

> Politicians go to university for a reason, so
> that they don't screw up international relations by saying something,
> well, 'unpolitical'. Peter, I really did think more of you.

I call a spade a spade. If you think the DIME approach is worth fighting
for, go ahead and do so. I'm just one person on this list, and others
may think your proposal has legs. To me, it doesn't seem like the right
approach.

> Note the insertions:
> 
> Peter Saint-Andre wrote:
>> Jonathan Chayce Dickinson wrote:
>>> Hey People,
>>>
>>> I was perusing over a couple of MSDN articles, when I came across DIME
>>> (Direct Internet Message Encapsulation). Jeez, I thought, that would fix
>>> all the file issues in Jabber (like SI). So I wrote a draft, any takers?
>>> What do you all think? Didn't want to push this forward until I got some
>>> feedback.
>>
>> Please don't waste our time with fanciful notions like this.
> 
> Really, ouch!

Hmm, yes, that was overstated. I'm sorry about that. But I did give some
reasons...

>> I say it is a fanciful notion because:
>>
>> 1. DIME was some Microsoft technology.
>>
>> 2. Which they submitted to the IETF in 2001 and never updated.
> 
> So what, they abandoned it, it remains viable. The gun was a huge advent
> in terms of war, it turned knives/swords obsolete, yet you still find
> them on guns in the form of a bayonet.

An Internet-Draft does not a sustainable technology make. There have
been thousands upon thousands of Internet-Drafts published. We need to
base our decisions on something more sustainable than that, I think.

>> 3. Which they then abandoned in favor of "SOAP Message Transmission
>> Optimization Mechanism" (MTOM).
> 
> Why do we have MTOM now? Probably DIME. Note: Optimization.

I don't understand what you mean by that. The W3C probably produced MTOM
for political reasons. :)

>> 4. Which in turn basically has nothing to do with XMPP or anything we
>> would work on here.
> 
> Which, in turn, basically has EVERYTHING to do with eXtensible (a.k.a
> XML, a.k.a SOAP, a.k.a Messaging, a.k.a *The Internet*) Messaging and
> Presence Protocol.

Gosh, SOAP is the world? I think not. :)

>> And I don't know why my name is on the protoXEP you wrote up, because I
>> had nothing to do with this.
> 
> Once again, sorry, maybe you should remove it from the template and put
> Romeo there instead as an example.

Well, I wrote the template. But I don't write everything that uses the
template.

> How frustrating. You try to help out: really, I wrote a whole freaken XEP.

Sorry you're frustrated. Do stick around so we can solve the file
transfer problem.

Again I think the fundamental question is: do we *really* need to stream
things, or can we use sender-upload and receiver-download for many or
most use cases, with in-band bytestreams as the fallback?

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 7338 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://mail.jabber.org/pipermail/standards/attachments/20070827/fd8de55c/attachment.bin>


More information about the Standards mailing list