minor #14888 [Messenger] More on unique Redis consumer names (danmichaelo)

javiereguiluz · javiereguiluz · commit ce2b086730fe · 2021-04-21T15:01:45.000+02:00
This PR was merged into the 5.2 branch. Discussion ---------- [Messenger] More on unique Redis consumer names I managed to miss the warning that the (strem,group,consumer) combination must be unique. I think it would make it less easy to miss if it was also mentioned in the Supervisor section, so I've tried to add a mention of it there. I also expanded the original warning block a bit. There are many different combinations of transports and ways to manage workers, so I understand that the docs should not delve too deep into each one, but I think it's worthwhile to cover the Redis + Supervisor combo quite well, since it's one of the easier combinations to get started with, which also can work really well. Let me know what you think. Related: #11869 #35358 Commits ------- e73e923 [Messenger] More on unique Redis consumer names
diff --git a/messenger.rst b/messenger.rst
@@ -626,7 +626,18 @@ times:
     process_name=%(program_name)s_%(process_num)02d
 
 Change the ``async`` argument to use the name of your transport (or transports)
-and ``user`` to the Unix user on your server. Next, tell Supervisor to read your
+and ``user`` to the Unix user on your server.
+
+If you use the Redis Transport, note that each worker needs a unique consumer name to
+avoid the same message being handled by multiple workers. One way to achieve this is
+to set an environment variable in the Supervisor configuration file, which you can
+then refer to in `messenger.yaml` (see Redis section above):
+
+.. code-block:: ini
+
+    environment=MESSENGER_CONSUMER_NAME=%(program_name)s_%(process_num)02d
+
+Next, tell Supervisor to read your
 config and start your workers:
 
 .. code-block:: terminal
@@ -1239,9 +1250,13 @@ claim_interval       Interval on which pending/abandoned    ``60000`` (1 Minute)
 .. caution::
 
     There should never be more than one ``messenger:consume`` command running with the same
-    config (stream, group and consumer name) to avoid having a message handled more than once.
-    Using the ``HOSTNAME`` as the consumer might often be a good idea. In case you are using
-    Kubernetes to orchestrate your containers, consider using a ``StatefulSet``.
+    combination of ``stream``, ``group`` and ``consumer``, or messages could end up being
+    handled more than once. If you run multiple queue workers, ``consumer` can be set to an
+    environment variable (like ``%env(MESSENGER_CONSUMER_NAME)%`)` set by Supervisor
+    (example below) or whatever service used to manage the worker processes.
+    In a container environment, the ``HOSTNAME`` can be used as the consumer name, since
+    there is only one worker per container/host. If using Kubernetes to orchestrate the
+    containers, consider using a ``StatefulSet`` to have stable names.
 
 .. tip::
 
