Reconcile QQ node dead during delete and redeclare #14241
Conversation
Co-authored-by: Péter Gömöri <gomoripeti@users.noreply.github.com>
|
thanks @LoisSotoLopez - this looks like it will do what we discussed a while back. I feel unsure about adding another key to the queue type state for this, mainly becuause we'd have to keep |
dac1a44 to
72a48e9
Compare
There was a problem hiding this comment.
I don't like having two keys with similar information (nodes) that will need to be kept in sync. I think we need to move the current nodes list value to the the #{node() => uid()} map format and handle the two formats in the relevant places, mostly in rabbit_queue_type. we do need a list of the member nodes in a few places but we could add a convenience functio: rabbit_queue_type:nodes/1 that takes a queue record and returns a list of member nodes. internally it could just call rabbit_queue_type:info(Q, [members]) and extract the result from that then update all places where we explicity use get_type_state to extract the member nodes.
In addition I think we need to put the use of nodes as a map behind a feature flag to avoid new queue records with nodes map values being created in a mixed versions cluster.
We are moving the functionality of getting the nodes/members of an amqqueue from the `amqqueue` module to `rabbit_amqqueue`. This goes in the line of previous PRs work towards reducing direct access to the `QueueTypeState`, such as rabbitmq#13905. Also, we will need to discretize different formats of the `nodes` entry in the `QueueTypeState`, to support both the previous one as a list of nodes and the new one as a map of nodes to Ra UIds. Doing so in a module such as `amqqueue`, which feels like an accessor module around the `amqqueue` record, doesn't feel right.
|
@kjnilsson Thanks for the suggestions. Just wanted to let you know we are working on this. Had some incident we had to take care of this week but I'll be pushing this PR forward next week. |
|
@LoisSotoLopez we have to ask your employer to sign the Broadcom CLA before we can accept this contribution (or its future finished version). It is about one page long, nothing particularly unusual or onerous. |
|
That commit below is just for showing current progress. Have been struggling to understand why a few of the remaining tests fail. |
1074165 to
35ef780
Compare
| @@ -791,6 +810,23 @@ recover(_Vhost, Queues) -> | |||
| ServerId = {Name, node()}, | |||
| QName = amqqueue:get_name(Q0), | |||
| MutConf = make_mutable_config(Q0), | |||
| RaUId = ra_directory:uid_of(?RA_SYSTEM, Name), | |||
There was a problem hiding this comment.
Figured out what I was doing wrong but I'm not sure what's the best approach to fix it. Let me recap and raise some questions.
So when a QQ is declared I am setting the #{node() => ra_uid()} map in the Queue Type State. I'll refer to those values in the map as NodeRaUids, as opposed to the UId associated to each QQ as a ra cluster ( the one generated by calling rabbit_quorum_queue:make_ra_conf), which I'll refer to as ClusterRaUid.
When a queue is deleted and re-declared the Queue Type state gets re-generated (no doubt about that because it's a new queue). Therefore between queue reincarnations the Queue Type state will change. However, the ClusterRaUid will not change for those nodes that were dead during the delete+redeclare.
I was checking, on member recovery, whether the ClusterRaUid, as retrieved with ra_directory:uid_of, was associated to the current node in the map of NodeRaUids. That's not right.
My current approach is storing the NodeRaUids map in disk, as it is done for ClusterRaId, and on recover compare that in-disk map with the one in the QueueTypeState.
My questions for you guys are: does that last paragraph sound right? and if it does, what API should I use to store that in-disk copy of the NodeRaUids map? Is there any ra_ module that I could use?
There was a problem hiding this comment.
If we already store ClusterRaId that way, I guess we can persist more metadata that the Ra members won't otherwise have/preserve.
There was a problem hiding this comment.
The ClusterRaUid gets persisted through ra_directory.erl which seems designed to strictly storing Ra cluster's specific pieces of related information. Maybe we just need a qq_nodes_uids dets for the sole purpose of storing those NodeRaUids. Or using Rabbit's metadata storage.
Edited: forget about this, found the error that led to this confusion.
|
@LoisSotoLopez sorry, any updates or feedback on the new CLA? We cannot accept any contributions without a signed CLA at the moment, and this change won't qualify for a "trivial" one, even if those get exempt in the future. We are trying to make it digital one way or another but there's a risk that the process will stay what it currently is (just a document to sign and email). |
Yes, sorry about not having this already addressed. We are on summer vacations right now so the people who will be signing it for the whole company won't be able to do it until next week. I would do it myself but can't due to a intellectual property clause on my contract that wasn't there the last time I signed the CLA. Will try to get it signed asap. |
Proposed Changes
This PR implements the suggested solution for the issue described in discussion #13131
Currently, when a QQ is deleted and re-declared while one of its nodes is dead, this dead node won't be able to reconcile with the new queue.
In this PR we add the list of ra UIds for the cluster to each node queue record, so that when a Rabbit node recovers a queue it will be able to detect the situation described above and properly reconcile .
Types of Changes
Checklist
CONTRIBUTING.mddocumentFurther Comments