Client side consumer recovery #1043

scottf · 2023-11-18T12:17:03Z

The PR addresses consumer recovery when the client is disconnected from a server or when the stream or consumer leader is affected by an outage. At least the following situations are covered

Durable Push Consumer
Ephemeral Push Consumer with a long enough inactive threshold to survive a server outage or disconnection
Ordered Push Consumer
Durable Pull Consumer when used with endless Simplification API
Ephemeral Pull Consumer with a long enough inactive threshold to survive a server outage or disconnection when used with endless Simplification API
Ordered Pull Consumer with endless Simplification

scottf · 2023-11-26T22:11:01Z

src/main/java/io/nats/client/impl/MessageManager.java

+        lastMsgReceived.set(System.currentTimeMillis());
+    }
+
+    class MmTimerTask extends TimerTask {


This was extracted from anonymous so the timer could be re-used. Reduce the number of objects created during error conditions since the object can be reused instead of a new one made.

scottf · 2023-11-26T22:11:50Z

src/main/java/io/nats/client/impl/MessageManager.java

            heartbeatTimer.schedule(heartbeatTimerTask, alarmPeriodSetting, alarmPeriodSetting);
+            updateLastMessageReceived();


above code block reflects extracted MmTimerTask and the ability to reuse existing timer

scottf · 2023-11-26T22:12:41Z

src/main/java/io/nats/client/impl/MessageManager.java

        }
    }

    protected void shutdownHeartbeatTimer() {
        synchronized (stateChangeLock) {
            if (heartbeatTimer != null) {
+                heartbeatTimerTask.shutdown();
+                heartbeatTimerTask = null;


if the task is null it won't be re-used.

scottf · 2023-11-26T22:13:08Z

src/main/java/io/nats/client/impl/NatsConsumer.java

@@ -230,7 +230,7 @@ public CompletableFuture<Boolean> drain(Duration timeout) throws InterruptedExce
   }

    /**
-     * @return whether or not this consumer is still processing messages. For a
+     * @return whether this consumer is still processing messages. For a


fixed grammar

scottf · 2023-11-26T22:14:30Z

src/main/java/io/nats/client/impl/NatsConsumerContext.java

@@ -68,7 +68,7 @@ public class NatsConsumerContext implements ConsumerContext, SimplifiedSubscript
            .replayPolicy(config.getReplayPolicy())
            .headersOnly(config.getHeadersOnly())
            .build();
-        subscribeSubject = originalOrderedCc.getFilterSubject();
+        subscribeSubject = Validator.validateSubject(originalOrderedCc.getFilterSubject(), false);


validating in the constructor because it comes through here from a different code path

scottf · 2023-11-26T22:14:47Z

src/main/java/io/nats/client/impl/NatsConsumerContext.java

        PullSubscribeOptions pso;
        if (ordered) {
            if (lastConsumer != null) {
                highestSeq = Math.max(highestSeq, lastConsumer.pmm.lastStreamSeq);
            }
            ConsumerConfiguration cc = lastConsumer == null
                ? originalOrderedCc
-                : streamCtx.js.nextOrderedConsumerConfiguration(originalOrderedCc, highestSeq, null);
+                : streamCtx.js.consumerConfigurationStartAfterLast(originalOrderedCc, highestSeq, null, null);


more descriptive name

scottf · 2023-11-26T22:43:55Z

src/main/java/io/nats/client/impl/NatsFetchConsumer.java

+    public void heartbeatError() {
+        stopped.set(true);
+        finished.set(true);
+    }


this handles failure in the middle of a fetch.

scottf · 2023-11-26T22:44:41Z

src/main/java/io/nats/client/impl/NatsJetStream.java

-                                             boolean isAutoAck
-    ) throws IOException, JetStreamApiException {
+                                             boolean isAutoAck,
+                                             PullMessageManager pmmInstance) throws IOException, JetStreamApiException {


allows me to use the existing PullMessageManager which has state, instead of making a new one.

scottf · 2023-11-26T22:45:48Z

src/main/java/io/nats/client/impl/NatsJetStreamImpl.java

        ConsumerConfiguration originalCc,
        long lastStreamSeq,
-        String newDeliverSubject)
+        String newDeliverSubject, String consumerName)


when the ephemeral consumer did not have a name, the server generated one. This gets passed in now since it's not part of the originalCc. Also renamed, hopefully more descriptive

scottf · 2023-11-26T23:07:20Z

src/main/java/io/nats/client/impl/OrderedPullMessageManager.java

+                resetTracking();
+                if (pullManagerObserver != null) {
+                    pullManagerObserver.heartbeatError();
+                }


this is just simplified - the manager doesn't really need to do all this stuff like it did with push ordered. Now we just let the observer know there was a problem and it handles it.

scottf · 2023-11-26T23:08:04Z

src/main/java/io/nats/client/impl/PullMessageManager.java

    @Override
    protected Boolean beforeQueueProcessorImpl(NatsMessage msg) {
-        messageReceived(); // record message time. Used for heartbeat tracking


now handled in trackIncoming

scottf · 2023-11-26T23:09:05Z

src/main/java/io/nats/client/support/Status.java

@@ -40,6 +40,7 @@ public class Status {
    public static String EXCEEDED_MAX_REQUEST_MAX_BYTES = "Exceeded MaxRequestMaxBytes"; // 409

    public static String BATCH_COMPLETED                = "Batch Completed"; // 409 informational
+    public static String SERVER_SHUTDOWN                = "Server Shutdown"; // 409 informational with headers


This is a new one to me. Handling it now.

scottf · 2023-11-27T14:55:05Z

src/main/java/io/nats/client/impl/NatsMessageConsumer.java

+
+    private void setupHbAlarmToTrigger() {
+        pmm.resetTracking();
+        pmm.initOrResetHeartbeatTimer();


whenever there is some sort of exception, instead of dying, just turn the hb alarm on. It will fire because there are no incoming messages. Do this instead of do it synchronously and also so we don't try doing the same thing that just failed right away.

scottf · 2023-11-27T14:58:31Z

src/examples/java/io/nats/examples/jetstream/NatsJsUtils.java

@@ -218,7 +218,7 @@ public static byte[] makeData(String prefix, int msgSize, boolean verbose, int x
            return null;
        }

-        String text = prefix + "-" + x + ".";
+        String text = prefix + "-" + x;


Just some example stuff, not production code, always tweaking things because I'm obsessive

scottf · 2023-11-27T15:00:24Z

src/main/java/io/nats/client/BaseConsumerContext.java

+     * Gets the consumer name that was used to create the context.
+     * @return the consumer name
+     */
+    String getConsumerName();


I decided to make the consumer name available for pull ordered consumers. This helps reporting, and underneath I will maintain the name on error. I originally didn't want the name exposed, but it's exposed on push ordered, and of course can be seen by management api/CLI. It's intended for visibility and reporting

scottf · 2023-11-27T15:01:10Z

src/main/java/io/nats/client/JetStreamManagement.java

+     * @return a JetStream instance.
+     * @throws IOException various IO exception such as timeout or interruption
+     */
+    JetStream jetStream() throws IOException;


I was tired of not being able to get a JetStream context from a JetStreamManagement context.

scottf · 2023-11-27T15:01:57Z

src/main/java/io/nats/client/impl/MessageManager.java

@@ -35,13 +38,13 @@ public enum ManageResult {MESSAGE, STATUS_HANDLED, STATUS_TERMINUS, STATUS_ERROR

    protected long lastStreamSeq;
    protected long lastConsumerSeq;
-    protected long lastMsgReceived;
+    protected AtomicLong lastMsgReceived;


Atomic is threadsafe. Might be overkill, but probably not.

scottf · 2023-11-27T15:08:11Z

src/main/java/io/nats/client/impl/NatsConsumerContext.java

        PullSubscribeOptions pso;
        if (ordered) {
            if (lastConsumer != null) {
                highestSeq = Math.max(highestSeq, lastConsumer.pmm.lastStreamSeq);
            }
            ConsumerConfiguration cc = lastConsumer == null
                ? originalOrderedCc
-                : streamCtx.js.nextOrderedConsumerConfiguration(originalOrderedCc, highestSeq, null);
+                : streamCtx.js.consumerConfigurationForOrdered(originalOrderedCc, highestSeq, null, null);


reflects name and signature (internal) change

scottf · 2023-11-27T15:08:50Z

src/main/java/io/nats/client/impl/NatsConsumerContext.java

            pso = new OrderedPullSubscribeOptionsBuilder(streamCtx.streamName, cc).build();
        }
        else {
            pso = unorderedBindPso;
        }

        if (messageHandler == null) {
-            return (NatsJetStreamPullSubscription) streamCtx.js.subscribe(subscribeSubject, pso);
+            return (NatsJetStreamPullSubscription) streamCtx.js.createSubscription(
+                subscribeSubject, null, pso, null, null, null, false, optionalPmm);


reflects need to pass existing PullMessageManager (pmm) to re-use with new subscriptions since pmm has state

scottf · 2023-11-27T15:10:03Z

src/main/java/io/nats/client/impl/NatsJetStream.java

+            }
+            else {
+                mm = pmmInstance;
+            }


this is where I reuse the pmm instead of making a new one.

scottf · 2023-11-27T15:10:38Z

src/main/java/io/nats/client/impl/NatsJetStream.java

@@ -565,7 +573,7 @@ private String lookupStreamSubject(String stream) throws IOException, JetStreamA
    @Override
    public JetStreamSubscription subscribe(String subscribeSubject) throws IOException, JetStreamApiException {
        subscribeSubject = validateSubject(subscribeSubject, true);
-        return createSubscription(subscribeSubject, null, null, null, null, null, false);
+        return createSubscription(subscribeSubject, null, null, null, null, null, false, null);


all these method calls reflect internal signature change. These are brand new subscriptions so don't have a pmm

scottf · 2023-11-27T15:11:11Z

src/main/java/io/nats/client/impl/NatsJetStreamImpl.java

+        consumerCreate290Available = impl.consumerCreate290Available;
+        multipleSubjectFilter210Available = impl.multipleSubjectFilter210Available;
+    }
+


implementation to make a JetStream context from a JetStreamManagement context

scottf · 2023-11-27T15:12:28Z

src/main/java/io/nats/client/impl/NatsMessageConsumer.java

    protected final int thresholdMessages;
    protected final long thresholdBytes;
+    protected final SimplifiedSubscriptionMaker subscriptionMaker;
+    protected final Dispatcher userDispatcher;
+    protected final MessageHandler userMessageHandler;


making the PullRequestOptions on the fly, so need to keep the original user options, etc. around.

scottf · 2023-11-27T15:15:58Z

src/main/java/io/nats/client/impl/OrderedMessageManager.java

+            try {
+                jsm.deleteConsumer(stream, actualConsumerName);
+            }
+            catch (Exception ignore) {}


So on failure of (push) ordered, I want to re-use the same consumer name, but it's likely that the original consumer I was using is still around, I don't know why the ordered error was triggered. But because you cannot just update an existing consumer to change deliver policy/start sequence, I have to make an entire new consumer. So I just make sure I delete the old consumer. The times its does exception will most likely be that it actually was deleted and deleteConsumer threw a JSApi exception

scottf · 2023-11-27T15:36:20Z

src/main/java/io/nats/client/impl/OrderedPullMessageManager.java

-                throw ise;
-            }
-        }
-    }


This was based on the Push version where I had to maintain the same subscription handle that was given to the user, but it's done differently here and this is just more complicated since the subscription handle is kept from the user because they are using the simplification handle.

scottf · 2023-11-27T15:38:17Z

src/main/java/io/nats/client/impl/PullManagerObserver.java

@@ -15,4 +15,5 @@

 interface PullManagerObserver {
    void pendingUpdated();
+    void heartbeatError();


needs this because the manager can't be responsible for restarts because the PMM is only managing messages, not the subscriptions / state getting messages.

… for ease of recognition

wallyqs

LGTM

scottf added 8 commits November 16, 2023 14:43

Better disconnection handling under simplification

3ca2b8e

Better disconnection handling under simplification

77c69f0

Better disconnection handling under simplification

b477e14

Better disconnection handling for ordered push consumer

94d2ab7

Fixing simple ordered fetch consumer / test to handle disconnect better

2275447

Adding file logging to help track things down.

d3acee4

Test data files got corrupted

3b168d4

Fixed pushed ordered recovery

8f2d46f

scottf mentioned this pull request Nov 24, 2023

NATS Push Consumer Client Stuck after NATS JetStream Node Failure nats-io/nats-server#4624

Closed

scottf changed the title ~~Robust simplification~~ Client side consumer recovery Nov 24, 2023

self review

8c8c0b6

scottf commented Nov 26, 2023

View reviewed changes

self review fixed code changed for debug

0493d06

scottf commented Nov 26, 2023

View reviewed changes

self review fixed code changed with search replace

8c7c8ee

scottf requested a review from piotrpio November 26, 2023 23:12

removing testapp to make the pr smaller

26d3aaa

scottf commented Nov 27, 2023

View reviewed changes

removing connection listener stuff, fix was not implemented this way

22ed0ee

scottf commented Nov 27, 2023

View reviewed changes

aroc naming

ee0fb9d

scottf commented Nov 27, 2023

View reviewed changes

scottf added 2 commits November 27, 2023 10:39

refacttoring the push handling so it looks like the new pull handling…

2c297b3

… for ease of recognition

better test impl

fa30d51

wallyqs approved these changes Nov 27, 2023

View reviewed changes

scottf merged commit a4b50a9 into main Nov 27, 2023

scottf deleted the robust-simplification branch November 27, 2023 23:45

This was referenced Nov 28, 2023

NATS Push Consumer Client Stuck after NATS Node Failure #997

Closed

Ordered Push don't cancel sub if trying to recover #1045

Merged

		heartbeatTimer.schedule(heartbeatTimerTask, alarmPeriodSetting, alarmPeriodSetting);
		updateLastMessageReceived();

Client side consumer recovery #1043

Client side consumer recovery #1043

Uh oh!

Conversation

scottf commented Nov 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scottf Nov 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

scottf Nov 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

scottf Nov 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

scottf Nov 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

scottf Nov 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

scottf Nov 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wallyqs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

scottf commented Nov 18, 2023 •

edited

Loading

scottf Nov 26, 2023 •

edited

Loading

scottf Nov 26, 2023 •

edited

Loading

scottf Nov 26, 2023 •

edited

Loading

scottf Nov 26, 2023 •

edited

Loading

scottf Nov 26, 2023 •

edited

Loading

scottf Nov 27, 2023 •

edited

Loading