[Standards] Proposed XMPP Extension: MAM Fastening Collation

Dave Cridland dave at cridland.net
Thu Jan 2 23:56:14 UTC 2020

On Thu, 2 Jan 2020 at 20:42, Marvin W <xmpp at larma.de> wrote:

> On 1/2/20 6:23 PM, Dave Cridland wrote:
> > It's not clear to me whether sending two poop emojis should count as two
> > reactions or just one. (Surevine once had an internal product that, by
> > accident, allowed "multilikes". We liked it. Often more than once)
> >
> > As things stand, we're effectively defining that people can like (or
> > poop emoji) other messages as many times as they like.
> As a reminder, https://xmpp.org/extensions/inbox/reactions.html
> specifically said that there can only be one of each reaction per user:
> "A <reactions> element MUST NOT contain the same reaction more than
> once. A receiving entity SHOULD ignore duplicate reactions inside a
> <reactions> element." This is a feature specifically wanted and desired
> by the authors, if MAM-FC is not supposed to be able to handle this
> feature, then Reactions can't make use fastening.
Would this work if we just counted unique fastening summaries for each
user, removing any duplicates?

It doesn't make much difference to me - it's just a unique constraint with
different fields in.

Interestingly that would show edits via fastenings as only ever having one
(which is probably OK), receipts etc would show a count per user but avoid
duplicates, which seems sensible.

Also trivial to include two counts, one with and one without duplicates,
which would satisfy either case.

> > In general, the "edit" case (as with any retraction case and others) is
> > a little shaky here because of authorisation. I'm very much open to
> > ideas here - it might be sensible to have a switch like the "external"
> > which pulled in the stanza from attribute, for example, or we might
> > simply choose to have servers "know" about edits (but see below).
> In the unfinished proposal I shared in the last mail, each <applied>
> also had a from attribute which is the from of the message that included
> the fastening. This however does not work in your model due to multiple
> fastenings being collapsed if they have the same type.
Yes, but that's needed if we want to support, for example, reactions (or
even "likes") in a large chatroom.

Sending 50+ likes for each message would be extraordinarily painful in
terms of bandwidth, and it's not unreasonable to assume that might happen
quite often.

This is massively amplified by receipts - it's very useful to know if
anyone, or everyone, has seen a message, but in a large chatroom that's a
lot of receipts.

> >
> >     Also note that the edit proposed in this ProtoXEP is not really
> >     compatible with the edit proposed in the fastening XEP
> >     <https://xmpp.org/extensions/xep-0422.html#external-payloads> as it
> is
> >     completely unclear what is supposed to happen with the <external
> >     name="body"/>.
> >
> >
> > Yes, quite, these need pulling into the fastening summary if they exist
> > (ie, they're not being pulled in from the encrypted blob). Totally happy
> > to make that clear.
> I think you missed the point that the <external /> is not a child of
> <edit /> but a sibbling. According to your proposal, only childs of
> <edit /> would be inside the <applied> element. So even pulling in the
> body doesn't describe how to handle this at all. This might as well be
> an issue in the XEP-0422 as it also says that an <apply-to> may contain
> several fastenings, but they must all be of the same type, and body
> doesn't seem to be the same type as <edit />
See last para of XEP-0422 §3.2

But in any case, the external things direct the client to the external
parts, which logically form part of the fastening.

> > FWIW, we use Chat Markers as super-receipts only, so I may have a blind
> > spot there.
> This pretty much confirms my feeling that this XEP is targeted for very
> few specific usecases which is exactly the opposite of generic. The
> whole purpose of fastening is to have something generic and that's the
> reason why we are delaying the proposed reactions XEP. If we now build
> something that is again not generic (and can't even handle the concept
> and ideas layed out in that proposed XEP) we won nothing and
> unnecessarily delayed a real feature that is often requested by end users.
I don't see a concrete suggestion here.

It may well be that Chat Markers don't make a suitable case for fastening,
and I'm fine with that.

> >
> >
> >     I also don't like that if I have a shell fastening, I only get to
> know
> >     that there are fastened messages. If there is 20 edits and one
> reaction
> >     and I want to know who send this reaction, I shouldn't be requires to
> >     download the 20 edit fastenings (using your proposed "fastenings" MAM
> >     call). I think all <applied> elements should carry the id of the
> >     underlying message from the archive, so that I can specifically
> >     fetch those.
> >
> >
> > Shell fastenings are for encrypted cases, so the server simply cannot
> > tell what these are and has no way to tell, by design. This was
> > specifically to satisfy cases where "everything must be encrypted".
> What I was proposing is that instead of adding the shell="true", add an
> id="mam-archive-id-of-the-fastening-message", so that clients can
> specifically fetch those instead of having to fetch all others (which
> may have not been shell fastenings). As shell fastenings don't have a
> count (at least I presumed that, because you'd just be counting the
> number of shell fastenings which sounds pretty useless), one shell
> applied element always references exactly one message.

Shell fastenings are pretty useless, yes. But having them individually
listed would, i think, be worse - and we can already pull the complete list
of them if we need to. It's an additional round-trip, certainly, but that's
a penalty if you're encrypting everything.

> > But if each individual fastening type has to have code support in the
> > server, that really slows down the rate of innovation.
> I am aware this is an issue because servers update slow. But rather than
> putting arbitrary restrictions of what can be done with fastenings, I'd
> prefer to have two levels of fastening collation, one that only
> describes the existence of fastenings (pretty much like shell fastenings
> in your proposal) and one that does smart collation. That second level
> would then only work if server supported that specific fastening type,
> so it could sum the reactions for reactions (including the limit of one
> reaction per kind), directly apply edits or whatever else comes in your
> mind. Of course both clients and servers would need to announce which
> smart fastening collations they support and servers should only use
> those that the client announced to support.
I don't think that's a sustainable path to take.

I'm not saying that having per-fastening code in every server is slow just
because there's a lag to them updating, I'm saying it'll be slow because
you'll have several round-trips between server developers and client
developers as the feature is developed and it'll take literal years for
each one to become stably deployed, as well as creating a vast amount of
workload for the XSF and its participants.

> > I'm absolutely not averse to having the client add in how a fastening
> > should be summarized - that was my original design actually - but we
> > decided that the count + latest was sufficient for all cases.
> I think this count thing is a bad idea. It is totally targeted towards
> reactions but even fails to provide the guarantees needed for proper
> usage of them. This is by the way getting way more complicated in
> semi-anon MUCs, because servers have to ensure that I cannot create an
> arbitrary number of likes by just changing the nick or changing the
> reactions someone else did by taking over their nick while they are
> offline, so we probably would need to incorporate occupant id somehow.
You're expecting servers to track individual authorisation of likes within

if it does, then it can filter entirely, surely? If it does not, then the
user's MAM archive cannot help anyway without an information leak from the
chatroom, can it?

> The latest thing is at least not a bad thing in most cases, so I'd be
> fine with that, as long as it is filtered per sender (showing the
> content of the latest of that fastening type for each sender instead of
> latest in total).

Again, if you have 50 likes in a chatroom that'd be a disaster. If your
client needs the individual likes, it can ask for them easily enough (if
you want only likes and not other fastenings that'd be trivial to add).

> > It *is* unfinished - of course. It doesn't even have a XEP number
> > (though it does have a schema etc already).
> This is perfectly fine of course. I was only mentioning this because we
> are already requiring everyone to use fastening even though we kind of
> agree that this isn't really a good idea at this stage because it is
> rather unfinished.
You seem intent on twisting my words, so allow me to return the favour.

You're saying that the XSF should only accept completely finished proposals
in this space?

> >     Just to come up with a new potential idea that would be hard to
> >     impossible to realize with your proposal: Clapping; on a popular blog
> >     platform you can clap for blog posts. Every user is entitled to up
> to 50
> >     claps on a blog post, but you can't undo claps. Usually only the
> total
> >     number of claps is displayed, but there is a detail view where you
> can
> >     see which persons clapped and how often.
> >
> >
> > Right, so, how would this work with MAMFC as written.
> >
> > So, users can clap on a message - OK, so we have a fastening type of
> > <clap/>. This summarizes to <clap/>, and has a full fastening of
> > <clap/>, so that's trivial for MAMFC so far.
> >
> > The total number of claps is shown, which again is fine for MAMFC.
> >
> > Our problem is that each user can only clap 50 times, so one assumes
> > that in a chatroom of 10 users, the maximum clappage can be 500? (or
> > 450, if one cannot clap oneself, which is at least gauche...). MAMFC
> > cannot currently enforce that, you'd have to enforce it client-side, or
> > by introducing a blog service which mediating clapping.
> >
> > But equally, it would have to be enforced somewhere, and to enforce it
> > inside MAM would mean it would have to be enforced in code on every
> > users' server who participated.
> >
> > If we ignore the limit, it becomes trivial - if we have "undo" claps,
> > then I think MAMFC should be able to handle that, though I've not
> > specified that yet because I was slightly ahead of Kev's XEP-0422
> > update. But I will do.
> >
> > So, summary: We cannot do your clap proposal with MAMFC, nor with any
> > similar system which doesn't involve either hardcoding the support in
> > every server or including an arbitrarily complex summarizer DSL code
> > fragment alongside every fastening.
> >
> > Is this a problem?
> If we picked a different MAM-FC the 50 per user (instead of an n*50 for
> n users) limit would entirely be possible: we don't have MAM-FC do any
> summing and we add the from to the <applied> and instead have each user
> always sends their total number of claps instead of just <clap /> for
> each single clap and replace the previous fastening. We then end up with
> a list of the latest number of claps of each user like this:
> <applied from="X"><clap count="20" /></applied>
> <applied from="Y"><clap count="40" /></applied>
> <applied from="Z"><clap count="60" /></applied>
> The element from Z has a too high number, so we just cut it to 50, the
> total count thus would be 110.
> What is not possible using this "latest" approach is that we can't
> verify that users are not removing claps. If we get rid of the "latest"
> mentality completely, we might see that X had send more claps before:
> <applied from="X"><clap count="10" /></applied>
> <applied from="Y"><clap count="20" /></applied>
> <applied from="Z"><clap count="30" /></applied>
> <applied from="X"><clap count="50" /></applied>
> <applied from="X"><clap count="20" /></applied>
> <applied from="Y"><clap count="40" /></applied>
> <applied from="Z"><clap count="60" /></applied>
> The client would then be able to calculate a sum of 140 for this which
> is the correct number under that desired specification. Of course this
> has a higher overhead. If the server has support for understanding
> clappings, they could just send
> <applied from="X"><clap count="50" /></applied>
> <applied from="Y"><clap count="40" /></applied>
> <applied from="Z"><clap count="50" /></applied>
OK, so you're saying that the client sees there are claps, and can retrieve
all the clap fastenings and process them locally.

Which can be done with this ProtoXEP.

> if both server and client announce support for smart summarizing of
> clappings, it could even become
> <clappings count="140" />
How does the server know what the legal limit is for claps, though? Have
you declared that every server in existence should support your limit of 50
and no take-backs? Otherwise this cannot possibly work.

As entertaining as this is, therefore, it seems to be a specious argument.

> The point I want to make and the problem here is that your restrictions
> for MAM-FC will effectively make certain usecases impossible. The only
> way to prevent that is to not make restrictions of fastenings that you
> don't understand. If you don't want fastening to be generic enough to be
> used for most usecases, fastening itself becomes useless.
Although MAMFC will work, it just may require an additional round-trip for
this contrived case.

> Yes, requiring server support is bad, but I think an approach is
> possible where the server doesn't need to support a fastening specific
> collation, it just can improve its bandwidth usage by doing so. Clients
> need to be able to support the non-summarized version anyway (because
> that's what they receive when not fetching from MAM) and they should
> also be interested in using smart summaries where possible to reduce
> bandwidth (thus downloading from MAM faster). Thus there is a natural
> wish of both clients and servers to implement the smart summaries when
> certain fastening types become popular whereas unpopular fastening types
> can still be used with no issues because you didn't put any unnecessary
> server side restrictions (and if they are unpopular anyways, the server
> doesn't need to fear bandwidth created by them not being summarized).

So there's a balance and compromise to be made, indeed.

I'm suggesting that MAMFC provides a compromise that works well enough for
most cases, and imposes an additional round-trip for some others.

I don't doubt it can be improved, but I think the fundamental shape of this

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.jabber.org/pipermail/standards/attachments/20200102/e78567bd/attachment-0001.html>

More information about the Standards mailing list