I fixed this

In T382948#10429035, @Ladsgroup wrote:

lol because it's the secondary dc and the cleaner script is not being run there and I assume these entries are not in codfw so they didn't get cleaned up by replication.

Codfw parsercache is properly clean.

In T382947#10429004, @Ladsgroup wrote:

But the biggest problem is that mediawiki core is quite lax on groups so there is quite a high chance that we will start serving production traffic from dbstore1008 and the other way around (dumps reading from other replicas and causing issues there as we have had this before many times)

But how are we going to approach a host going down here? Plus, if we change hostnames/hosts, we'd need to edit MW?

We should make them run with weight 0 in the general traffic set up, @Ladsgroup that means MW won't check for lag, am I right? (which I am fine with). I just want to make sure we don't send ANY general production traffic there.

Process still running - I think connection/credentials wise we are all set.

I have restarted phabricator_task_dump.service and based on strace I believe it is working

I have fixed this by changing the IP.
I've deleted the entries with the old IP

In T378940#10396116, @Marostegui wrote:

I will upgrade db2236 to 10.11.10 next week if all goes well and then we can close this task (10.11 isn't in the repo).

Ready for DC-Ops

This is ready for DC-Ops

This was a test to see if the change at https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/7 was applied.

Are there any significant schema changes on this upgrade?

In T370266#10156784, @LSobanski wrote:

Would there be interest in getting a DB snapshot (replica on a real-size host) to test this on? We took this approach when upgrading VRTS: T355541: Install a temporary DB host in m2 to support VRTS migration.

This is done

Weight was 300

This is all done - all hosts have now decommission tasks which are just pending on-site steps.

This is ready for DC-Ops

Ready for DC-Ops

This is ready for DC-Ops

Downtime removed, notifications enabled and the host being slowly automatically repooled.
Thanks everyone who responded to this incident! Much appreciated.

In T382744#10426163, @Marostegui wrote:

For the record:

for i in `db-mysql db2123 information_schema -e "select table_schema from tables where table_name='pagelinks';"`; do echo $i; db-mysql db2123 $i -e "set session sql_log_bin=0; alter table pagelinks engine=innodb,force; alter table recentchanges engine=innodb,force;";done

This is ready for DC-Ops