JAMES-4156 ADR: Deleted message vault single bucket usage

[Arsnael]

No description provided.

[mbaechler]

I would expect src/adr to be about James architecture.
If I understand correctly, this one is about Delete Message Vault extension.
If so, maybe introducing a way to split extensions ADRs from the main codebase would help limit the amount of ADRs one has to read to understand James?

[chibenwa]

+1 on the overall goal to keep James understandable.

On my side a conflicting goal come to mind: discoverability. Having a single folder for ADRs makes it a single place to speak architecture and at least I know where to look.

There might exist nice solution to conciliate both goals (like referencing new ADR locations in James main ADR location like establishing mailbox/src/adr ?) - maybe other projects using ADRs have already soved this problem.

Also would we intend to relocate the concerned ADRs?

IMO it would be a awesome mailing list topic!

[Arsnael]

@mbaechler Not against it, but I agree with @chibenwa on the fact that then it should be discussed with the community (ML) and should be treated in an other PR when we reach a proper consensus

If we start having adrs all around the place though I'm afraid it might be confusing. What about subfolders instead?

src/
  - adr/
    + core/
    + extensions/
[...]

I think it's clear, it's still in one place, subfolders allow easily for someone to just look at the core James ADRs if he just wishes to understand James basic core concepts, and also still easy to find extensions ADRs if one needs them? It would be confusing IMO to go look into the module of those extensions to find potential ADRs related to it.

? Will start a ML anyways on this today :)

[quantranhong1999]

it should be discussed with the community (ML) and should be treated in an other PR when we reach a proper consensus

+1

[Arsnael]

ML started

[mbaechler]

@Arsnael your proposal seems nice

[Arsnael]

Hmmm seems my mail did not reach the server-dev ML for some reason sorry... Will see

[Arsnael]

Alright mistake on the destination first time (I changed laptop sorry)... Now the ML is created.

@mbaechler are you ok if we merge this ADR now and reorganize later when the consensus is reached via the ML discussion? :)

[jeantil]

The pattern of using several "buckets" exists in several places ( and caused me a lot of pain for message queues ) I wonder if it ever was such a good idea and if the scope of the ADR shouldn't be expanded to a more general definition...
(which would also make the ADR relevant for core and make the whole discussion on splitting irrelevant)

[chibenwa]

The pattern of using several "buckets" exists in several places ( and caused me a lot of pain for message queues ) I wonder if it ever was such a good idea and if the scope of the ADR shouldn't be expanded to a more general definition...
(which would also make the ADR relevant for core and make the whole discussion on splitting irrelevant)

I see the point.

We could actally then consider this a rework of "how the bucket is encoded at the S3 level"

It "would then just be a S3 concern" and could likely be implemented as a prefix (instead of encoding a James bucket as a S3 bucket).

• + no code changes in other places, single bucket etc
• - migration is not trivial.
  • if we use `` (empty) as the default 'prefix' it could lead to conflict in the hierarchy (though we operate atop a well identified set of blboIdFactories so there might be tricks...)
  • But using a non empty default (james) bucket prefix means migration for significant amount of data
  • A fallback implemented at the object store level COULD be valid functionnally speacking but not acceptable from a performance perspective.
  • We could however dis-anbiguate things by moving non default buckets in a known prefix.

=> No need of migration for #default (mailbox, mailqueue)
=> james-uploads are not performance-important so do the deleted message vault so a fallback is enough awaiting expiration / data drop (which is ok)...

So I would propose:

[all #default at the base of the hierarchy]
buckets/jmap-uploads
buckets/deleted-message-vault-2025-12
buckets/deleted-message-vault-2025-11
...

(GC would then skip buckets/)

Side benefits: refactoring the bucket name translation, and the bucket auto-creation, which is a mess.

(which would also make the ADR relevant for core and make the whole discussion on splitting irrelevant)

The start of the discussion becomes irrelevant but not the discussion itself.

BTW One might also argue "I want to undersntand James while I am not using S3".

[jeantil]

BTW One might also argue "I want to undersntand James while I am not using S3".

then skip the S3 related ADRs but they are still part of the core ADRs :P

The start of the discussion becomes irrelevant but not the discussion itself.

I don't know : ADRs should represent architecturally structural decisions, if an ADR only impacts a single extension/module then it's an implementation detail not an architectural decision and it should probably be documented along the corresponding extension.

[chibenwa]

#2904 is a tiny step toward implementing "James buckets" as "object prefixes".

[Arsnael]

Closing related changes to deleted message vault for a lack of consensus reason. Moving work to linagora tmail.

[jeantil]

I think there is a misunderstanding :

• I don't think this requires an ADR but I do agree it's a worthwhile change
• I didn't completely agree with Benoit's proposed implementation direction and reminded him of an alternative which already exists in the code base and encodes the virtual path/virtual folder in the blob id instead of changing the bucketname

Note that my only concern with Benoit's change was that it resulted in a breaking change in the BucketNameResolver api contract.
The change didn't break code in james but since this is a public contract which has been published in official releases we must follow the process for introducing breaking changes. I don't even dispute the point that it is unlikely to actually have been used, but there is no way to know for sure.
Benoit closed the MR at this point and you are doing the same saying you will move the development to your closed source extension, which is your right but also a bit sad.

[chibenwa]

Hello @jeantil

I didn't completely agree with Benoit's proposed implementation direction and reminded him of an alternative which already exists in the code base and encodes the virtual path/virtual folder in the blob id instead of changing the bucketname

While conceptually better this direction indeed have broad impact; I must confess it do not seem like a reachable goal to consensually achieve a full refactoring of S3BlobStoreDAO to operate off a single bucket.

Marginal gains compared to just "rewritting the Deleted Message Vault" to use a single bucket do not seem in favor of the full-s3-refactoring idea.

the virtual path/virtual folder in the blob id instead of changing the bucketname

This is somewhat what #2902 achieves, and is in line with the current ADR.

BTW this is the direction we (linagorians) chose to move on with for Twake Mail.

you will move the development to your closed source extension, which is your right but also a bit sad.

I think there's a misunderstanding here indeed.

As employee I am accountable for time spent on the project and that it results in fixing the problem my employer faces.

If to reach this end this means moving forward on a separate, company owned repo then be it (Twake mail is not closed source, but AGPL v3 and owned by LINAGORA).

But indeed this is a problem others in the James community will face and indeed maintenance costs will be lower with this code change within James. And carrying the two approaches are not mutually exclusive, and could even converge later depending on feedback !

I propose to bring this to the mailing list thread, and to a vote:

• Address the use of several bucket of the Deleted Message Vault ( this ADR + JAMES-4156 refactoring Blobstore deleted message vault for appending messages into single bucket #2902 )
• Do not attempt to merge everything into a single bucket at the S3 level. We won't attempt to solve the issue globally at the S3 level, but focus on the scope where it is currently problematic, just within the scope of the "deleted message vault" which is where it is problematic for distributed + PG users.

Cheers,

Benoit

[jeantil]

I propose to bring this to the mailing list thread
👌