As far as I'm aware, this was fixed.

In T359472#10412624, @HCoplin-WMF wrote:

Note: This work is expected to be done by April. Should be minor change; see other linked patches for examples.

Note: This work is expected to be done by April.

I 'll switch to low, we can keep monitoring the next few weeks and see how this pans out.

@Nikki thanks for the explanation. As you note yourself, speed wise, there are many ups and downs.

Just processed those edit requests.

For the record, I just found another set of templates we want to update on most of the affected files (Cc-by(-sa)-layout should bypass the SDC_statement_exist template) and filed edit requests for them.

@LucasWerkmeister how was the 10 year estimation calculated?

This type of Kafka Consumer lag for that job isn't unheard of. In fact, just recently we had way higher consumer lags for commons specifically.

Indeed, it looks like the refreshLinks_partitioner rule is easily keeping up with the "upstream" rate of new jobs [0] but the "real" refreshLinks rule on partition 3 (commons) has a rather deep backlog.

That’s great, thank you!

I have processed those editrequests, so at least when a page contains multiple license templates, it will only need to be reparsed once.

Currently, the number of links to Template:SDC statement has value (as counted by search – this lags somewhat behind the “real” number, as it depends on a further job, but it should be a decent approximation)

Per IRC discussion, marking as High priority. @AntiCompositeNumber reports that this results in category changes being slow to propagate (Category:Johann Baptist Hops not showing up in Category:Hops (surname) yet).

Monitoring for sustained latency impact on low-traffic jobs is now live.

Since the three job types critical to uploads have now been moved to dedicated consumers (T379035), the primary follow-up here is monitoring (T378609) to detect when these kinds of isolation failures occur so that we can reactively isolate the "antagonist" job.

Thanks for flagging, all. Yes, this looks like another isolation failure on the low-traffic consumer, and appears to have largely self-resolved as of ~ 14:50 UTC on the 10th. I'll follow up on T379462 for this particular instance, and aim to prioritize T379035 when I'm back this week.

Happened to me some months ago.

Possibly the cause of T379476?

Subscribing myself. I'm seeing this for a while, and yes, today some mw tags are taking forever to update.

That said, it does seem like the p99 for AssembleChunkUpload jobs has spiked to ~15 min for the last 2 hours (was fine before that point), so maybe that is just it. Maybe driven by a spike in ChangeDeletionNotification jobs. Sounds like a dedicated queue as Scott suggests would really help.

@MBH lets open a separate new task to investigate, as the cause could be something different than the job queue thing this task is about. If you want you could email the HAR file to me ( bawolff@gmail.com ).

Now files are waiting ~10 minutes before publishing and doesn't published due to errors "Unknown server error" and "Incorrect CSRF token". Uploading (first UploadWizard step) was very slow too with the same behavior than in previous case: 3 files in queue and all other files waiting, after several minutes this 3 files uploaded and next 3 files in queue.

@Bawolff The problem described in T378276 is rised again. I have recorded a HAR file, please, give me an e-mail where should I send it. I will not clear any cookies because I don't know how to do it, I'll just seng you a raw file.

Thanks, @Bawolff - Yes, indeed, those both fan into the low-traffic consumer. While we don't really have a prioritization mechanism in this context that I'm aware of, it would probably be fairly straightforward to at least move them out of low-traffic to a dedicated consumer, as @Ladsgroup points out. I've opened T379035 to look into that.

It shouldn't be too hard to give it a dedicated lane with small concurrency

Just as an aside, I believe PublishStashedFile AssembleUploadChunks are considered low traffic job. Unlike normal jobs these are very latency sensitive, as they don't happen in the background, but the UI actually makes users wait well these jobs complete (See also T378276). It would be really great if somehow these jobs can be prioritized in a job queue backlog situation.