[Standards] Proposed XMPP Extension: MAM Fastening Collation

Dave Cridland dave at cridland.net
Wed Jan 8 13:02:21 UTC 2020

Heavy snipping. Not ignoring the other parts, just not answering them for
now at least.

On Mon, 6 Jan 2020 at 23:54, Marvin W <xmpp at larma.de> wrote:

> On 1/3/20 12:56 AM, Dave Cridland wrote:
> > Sending 50+ likes for each message would be extraordinarily painful in
> > terms of bandwidth, and it's not unreasonable to assume that might
> > happen quite often.
> I agree that we want to have summarized reactions for chatrooms, but I
> am not that set on extending summarizing to all fastenings with the same
> concept. IMO, if we can't come up with an example where this counting is
> actually a good idea other than reactions, it shouldn't be part of
> fastening (the generic concept) but part of reactions (the only
> realization needing that kind of summary).
It's not entirely clear to me that summarizing to a count (or multiple
counts) actively does harm, though. Yes, the count (or one of them) is
directly useful for Reactions and any similar lightweight interactions. But
for cases where we need to (eventually) fetch everything, we have that
covered as well.

> > This is massively amplified by receipts - it's very useful to know if
> > anyone, or everyone, has seen a message, but in a large chatroom that's
> > a lot of receipts.
> > [...]
> > It may well be that Chat Markers don't make a suitable case for
> > fastening, and I'm fine with that.
> I don't think you can get the information that "everyone" in a larger
> group chat has seen a message, because you don't know what the
> recipient's clients actually do, they might just not support read
> receipts or have disabled them for privacy reasons. The only really
> sensible thing you can know, is if the group chat server received the
> message, and that's already given if you fetch it from MAM.
There are cases where you know everyone has read the message, and there are
cases where everyone has read the message but you don't know.

> I haven't noticed that part, but it seems to be an issue in 422: if you
> want to have two fastenings of the same type at the same time, they
> can't have externals, because else you wouldn't know which external
> belongs to which fastening.

This is an excellent point.

> > Shell fastenings are pretty useless, yes. But having them individually
> > listed would, i think, be worse - and we can already pull the complete
> > list of them if we need to. It's an additional round-trip, certainly,
> > but that's a penalty if you're encrypting everything.
> I was proposing to list the archive id of each fastening, not the
> content. The reason for this is that I don't want to be required to pull
> all fastenings every time there is a change for a message. If I have the
> archive id on all the fastenings, I can easily see which of those I
> already downloaded in the past/have in my local cache and only download
> the new ones.

Ah, I think I understand your use-case, but I think it's also covered -
especially with the additions we've already agreed.

You know the latest archive id for which you pulled the fastenings - that's
in the (new to MAMFC) latest tag in the MAM <fin/>. You've asked for the
archive id (and some other details, all of which I've agreed) to be
included in the latest fastening information as well, which optimises this

You can ask for just fastenings for a message (MAMFC current), for a
particular fastening type (agreed change earlier in this thread) and filter
for messages after an archive id (RSM+MAM current).

So I think this accomplishes everything you're asking for, except that
obtaining the full fastenings for a message always requires a round-trip
(unless you're not trying to use MAMFC at all, in effect, and ask for

Note however that including only the latest fastening in detail means it's
also easy to look at the MAMFC summary and decide whether there could be
new fastenings to fetch, as well (especially with the additions you've

So this *is* adding a RTT per message which has new fastenings *if* you
care about that fastening summary, but including all the fastenings into
each result is mere collation which the client can do itself just by asking
for everything anyway.

> Yes, encryption makes some things harder. I'd still like to make things
> with encryption as good as possible and not improve the unencrypted case
> on cost of the encrypted case just because the encrypted case can never
> be perfect.

That's not the intent. The intent is not to assume we always use whole
stanza encryption, and sometimes use either content-only encryption or even
no E2EE at all. Following from that (and I note the latter two cases are
equivalent for MAMFC I think), we aim for graceful degradation as the
server loses visibility on the data.

> > I'm not saying that having per-fastening code in every server is slow
> > just because there's a lag to them updating, I'm saying it'll be slow
> > because you'll have several round-trips between server developers and
> > client developers as the feature is developed and it'll take literal
> > years for each one to become stably deployed, as well as creating a vast
> > amount of workload for the XSF and its participants.
> How is this worse than having a "generic" feature that works good for
> reactions and edits, but doesn't work for future extensions and thus
> will need to be updated every now and then, having the same cycle you
> described but at the same time may break all the existing infrastructure
> for previous fastenings.
If MAMFC ends up that way - and I don't think it will - then it's started
with a reasonably solid solution for all the cases of fastenings we've
identified and then needed additions subsequently, rather than started with
a suboptimal solution from the outset. In the worst case, the long-term
effort ends up equal.

> Another issue that I noticed: you are relying on the thing you call
> 'fastening summary'. What happens if we allow for more complex reactions
> that can't be put as an attribute, but are realized as an child element,
> like custom images. This was already expressed as a desired feature here
> on the list and is easy to realize from the reactions proposal in the
> inbox, but hard using the "hidden" reactions proposal in your (Proto)XEPs.
Isn't this trivial? Just have an attribute in the fastening itself which is
a hash (or unique, stable identifier) of the custom image, and it'll
summarize neatly.

Perhaps this is where our mutual confusion lies? I see the combination of
XEP-0422, this specification, and future fastening design as a largely
green field thing, and we'd design fastenings such that the rules in MAMFC
provide useful results for them, rather than designing anything we want
without regard to the status quo and then having to adjust MAMFC to fit?

> > You're expecting servers to track individual authorisation of likes
> > within chatrooms?
> I'd expect clients to track reactions in group chats by occupant id
> (XEP-0421) instead of senders nickname. This was already teased in the
> reactions proto xep
> <https://xmpp.org/extensions/inbox/reactions.html#acceptable-reactions>,
> although occupant ids weren't a thing back then.
> Thus I'd expect servers to do the very same. For edits, that's even a
> stronger case and I would hope for a server at some point to just filter
> out invalid edits. Even if you don't make that part of the XEP it would
> be a sensible idea for servers to do that to reduce load for both server
> and clients at no practical cost (as clients would have to filter them
> out anyway).
Well, if the chatroom does this then fine, but I think as long as we handle
XEP-0421 and include a count per-user as well as total we're covered here
in terms of the UX.

> > OK, so you're saying that the client sees there are claps, and can
> > retrieve all the clap fastenings and process them locally.
> I wasn't talking about the client retrieving all fastenings, in this
> example the <applied> elements are directly in the <result>, the same as
> in your ProtoXEP. It doesn't need another round trip to request anything.
Right, but this represents no difference to doing no collation at all, and
just including these directly in MAM. Which we don't want to do because it
becomes painful.

Or, we end up with unbounded length stanzas, which is also painful.

> > Although MAMFC will work, it just may require an additional round-trip
> > for this contrived case.
> As mentioned above, I see the risk of the summarizing - if done wrong -
> to cause the client to be unable to see that they need to fetch
> something from the server. And we want clients to only load the minimum
> necessary.
No, I don't think we're ever in that position.

I think we're in a position whereby should a client want to obtain a list
of fastenings for a given message, it always knows if it has the full list
already, or if it needs to fetch them, but to fetch will be (at least)
1*RTT (interleavable). This means that to display a screenful of messages
and have the client obtain all fastenings for all messages displayed will
be 2*RTT - but a client can elide that if it has the fastenings already. We
also have simple mechanisms to reduce the amount of data to (more or less)
precisely what the client wants.

The only way I see to reduce this to 1*RTT would mean having all fastenings
in full in the result, which would make results huge, so seems unacceptable
to me. It would also be a lot of data, which we could reduce by the client
having detailed negotiation - but that seems to add a lot of complexity.
The summary detail in each result would also vary depending on the extent
of the RSM, which is also quite complex on the server.

> > I'm suggesting that MAMFC provides a compromise that works well enough
> > for most cases, and imposes an additional round-trip for some others.
> >
> > I don't doubt it can be improved, but I think the fundamental shape of
> > this works.
> I am certain it will work. The same applies to other possible ideas.

The enemy of the good is the dream of the perfect. :-)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.jabber.org/pipermail/standards/attachments/20200108/170795c2/attachment.html>

More information about the Standards mailing list